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PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  under  the 
task  order,  Defense  Modeling  and  Simulation,  and  responds  to  an  objective  of  the  task,  to 
“provide  technical  support  to  operational  activities  of  DMSO,  e.g.,  assessing  the  benefits 
achieved  from  use  of  modeling  and  simulation.”  The  work  was  sponsored  by  the  Defense 
Modeling  and  Simulation  Office  (DMSO),  Office  of  the  Director  of  Defense  Research  and 
Engineering. 

IDA  research  staff  members  and  military  and  civilian  personnel  from  Department 
of  Defense  Components  were  members  of  the  Modeling  and  Simulation  Benefits  Task 
Force  set  up  by  the  DMSO  Modeling  and  Simulation  Working  Group.  The  initial 
objective  was  to  retrieve  documented  accounts  of  the  benefits  of  modeling  and  simulation. 
The  Task  Force  conducted  its  collection  effort  during  the  period  of  March  to  September 
of  1995. 

The  efforts  and  contributions  of  the  individual  members  of  the  Modeling  and 
Simulation  Benefits  Task  Force,  under  the  direction  of  LtCol  David  A.  Bartlett,  United 
States  Marine  Corps,  are  gratefully  acknowledged.  The  following  IDA  research  staff 
members  reviewed  this  document:  Dr.  Richard  J.  Ivanetich  and  Dr.  Jesse  Orlansky.  Their 
contributions  are  also  acknowledged. 
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EXECUTIVE  SUMMARY 


The  Modeling  and  Simulation  Working  Group,  under  the  auspices  of  the  Defense 
Modeling  and  Simulation  Office  (DMSO),  authorized  the  formation  of  a  Modeling  and 
Simulation  Benefits  Task  Force  (MSBTF)  to  capture  documented  reports  of  the 
quantifiable  benefits  of  modeling  and  simulation  (M&S).  This  effort  is  an  initial  step  in 
fulfilling  Sub-objective  6-1  of  the  Department  of  Defense  Modeling  and  Simulation 
Master  Plan,  to  quantify  the  impacts  of  M&S,  with  the  results  incorporated  into  the 
longer-range  Impact  Assessment. 

The  MSBTF  first  met  February  23,  1995,  continued  meeting  monthly,  and  stood 
down  on  September  21,  1995.  This  report  describes  the  Task  Force’s  efforts  and 
findings. 

During  the  Task  Force’s  “data  capture”  period,  responses  were  received  from  two 
Requests  for  Information  (RFI),  Task  Force  members  collected  inputs  from  their 
respective  Services  and  Agencies,  and  formal  studies  were  collected  from  Federally 
Funded  Research  and  Development  Centers  (FFRDCs)  and  Department  of  Defense 
(DoD)  Components.  The  RFI  conducted  for  the  Task  Force  received  12  responses.  A 
separate  RFI  conducted  for  the  test  and  evaluation  community  received  15  responses. 
Overall,  Task  Force  members  collected  over  90  contributions. 

While  no  formal  assessment  could  be  accomplished  based  on  the  relatively  small 
amount  of  information  gained  in  a  limited  period  of  time,  an  informal  meta-analysis — ^an 
analysis  of  other  organizations’  analyses — is  included  in  this  report.  Meta-analysis  has 
two  serious  shortcomings:  it  is  based  on  information  often  a  year  or  more  old,  and  it  is 
based  on  information  typically  biased  towards  the  positive.  The  positive  bias  stems  from 
the  tendency  to  underreport  negative  results  and  the  proponent’s  natural  desire  to  share 
positive  results.  The  Task  Force  provided  some  old  and  some  very  new  inputs.  FFRDC 
inputs,  while  recently  published,  were  sometimes  based  on  data  collected  in  prior  years. 

Findings 

The  applications  of  M&S  to  acquisition  are  many.  Twenty  case  studies  of  Target 
Interaction,  Lethality  and  Vulnerability  showed  a  30-to-l  return  on  investments  in  M&S 
support  for  milestone  decisions  and  the  Cost  and  Operational  Effectiveness  Analysis 
process.  The  Army  Missile  Systems  Command  reported  a  total  of  over  $320  million  in 
cost  avoidance  or  savings  from  10  case  studies.  Eight  case  studies  were  provided  from 
the  Virtual  Proving  Ground,  U.S.  Army  Test  and  Evaluation  Command.  Two  similar 


events  were  conducted  for  Apache  Longbow  Force  Development  Test  and 
Experimentation,  one  using  extensive  simulation  and  the  other  using  physical  equipment. 
The  simulation-supported  event  executed  twice  as  many  trials,  with  fewer  personnel,  in 
less  time,  at  lower  risk  to  personnel,  for  $700,000  versus  the  $4  million  spent  in  using 
physical  equipment. 

Training  applications  of  M&S  were  commonly  used  and  the  results  were  positive. 
Reporting  was  thorough  on  individual  skills  training,  including  both  cognitive  and 
psychomotor  skills.  Cognitive  skills  trainers,  typically  computer-aided  instruction,  paid 
for  themselves  in  five  years  or  less.  Psychomotor  skills  trainers,  e.g.,  flight  simulators, 
driver  trainers,  conduct  of  fire  trainers,  and  maintenance  trainers,  were  all  shown  to  be 
cost  effective  when  properly  mixed  with  training  on  the  real  equipment.  At  this  level, 
analysts  have  well-established  theories  and  experimental  methods  for  conducting 
analysis.  The  same  is  not  true  for  unit  training,  particularly  of  high  echelon  units.  The 
high  cost  of  a  Joint  or  Combined  exercise  precludes  the  repeated,  controlled  experiments 
necessary  to  gather  meaningful  data  on  the  benefits  of  additional  learning  trials. 
However,  multi-million  dollar  savings  are  reported  when  comparing  computer-assisted 
command  post  exercises  to  field  training  exercises. 

Although  M&S  are  used  extensively  in  analysis,  few  reports  documenting  benefits 
were  found  by  the  Task  Force.  No  reports  were  received  claiming  cost  savings  due  to  the 
application  of  M&S  to  analysis.  Information  provided  by  members  of  the  analytic 
community  suggests  that  they  tend  to  measure  the  effect  of  their  analysis  on  the  decision¬ 
making  process  rather  than  the  effect  of  M&S  on  their  analysis.  The  effect  of  M&S  on 
analysis,  while  real,  is  problematic  to  quantify.  In  contrast,  the  training  and  acquisition 
functional  areas  were  more  likely  to  gather  quantitative  data,  specifically  cost  data. 

Across  functional  areas,  measures  of  effectiveness  (MOEs)  were  not  universally 
agreed  upon.  Consistency  within  functional  areas  was  somewhat  better.  Analytic 
frameworks,  including  MOEs,  need  to  be  developed  and  applied  consistently  throughout 
DoD.  Frameworks  should  be  unique  to  each  functional  area  and  perhaps  even  functional 
sub-areas.  As  analytic  frameworks  are  developed,  they  should  be  disseminated  as 
recommended  best  practices. 

Finally,  a  formal  reporting  mechanism  does  not  exist  for  gathering  information, 
nor  do  the  methodologies  exist  for  objectively  assessing  the  value  of  using  M&S.  If  the 
effects  of  M&S  are  to  be  collected,  assessed,  and  disseminated,  then  methodologies 
should  be  determined  and  reporting  pipelines  should  be  established  from  the  M&S 
developer  and  user  communities  through  the  DoD  Components  to  DMSO. 
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1.  INTRODUCTION 


The  Modeling  and  Simulation  Working  Group,  created  under  the  direction  of  the 
Defense  Modeling  and  Simulation  Office  (DMSO),  authorized  the  formation  of  a 
Modeling  and  Simulation  Benefits  Task  Force  (MSBTF)  to  capture  documented  reports 
of  quantifiable  benefits  of  modeling  and  simulation  (M&S).  This  effort  is  an  initial  step  in 
fulfilling  Sub-objective  6-1  of  the  Department  of  Defense  Modeling  and  Simulation  Master 
Plan  [DMSO  1995],  to  quantify  the  impacts  of  M&S,  and  will  be  incorporated  into  the 
longer-range  Impact  Assessment.  This  report  describes  the  Task  Force’s  efforts  and 
findings. 

The  MSBTF  first  met  February  23,  1995,  continued  meeting  monthly,  and  stood 
down  on  September  21,  1995.  The  data  collection  effort  occurred  during  the  period  of 
March  1995  to  September  1995. 

1.1  TASKING  FROM  DOD  M&S  MASTER  PLAN 

Department  of  Defense  Modeling  and  Simulation  Master  Plan  [DMSO  1995] 
contains  six  broad  objectives.  Objective  6,  “Share  the  benefits  of  M&S,”  the  basis  for  the 
Task  Force’s  formation,  has  three  sub-objectives: 

•  6-1:  “Quantify  the  impacts  of  M&S.” 

•  6-2:  “Educate  potential  M&S  users  (DoD,  Congress,  other  government 

agencies,  industry).” 

•  6-3:  “Support  bi-directional  technology  transfer.” 

Sub-objective  6-1  further  states  that  to  quantify  the  impacts  of  M&S,  the 
following  actions  must  be  accomplished: 

•  Collect  and  analyze  data  from  ongoing  efforts,  planned  experiments,  and 
demonstrations  to  assess  the  impacts  of  M&S. 

•  Develop  quantitative  measures  of  the  benefits  of  M&S  (e.g.,  readiness  impact, 
cost  savings,  and  effectiveness)  to  (1)  allow  assessment  of  the  utility  of  M&S 
and  (2)  support  investment  decisions. 


•  Establish  the  DoD-wide  impact  of  M&S. 

DMSO  has  initiated  three  activities  in  response  to  this  tasking: 

•  The  Institute  for  Defense  Analyses  (IDA)  has  collected  information  on  48 
projects  and  17  activities  sponsored  by  DMSO  in  fiscal  years  1992  through 
1994  [Hammond  et  al.  1995]. 

•  The  MSBTF  has  conducted  an  initial  data  capture  on  benefits  from 
Department  of  Defense  (DoD)  Components,  the  results  of  which  are  reported 
here. 

•  A  sustained,  long-range  Impact  Assessment  has  begun,  employing  the 
previous  efforts  as  input. 

1.2  TASK  FORCE  TIMELINE 


Table  1.  M&S  Benefits  Task  Force  Timeline 


Task  Force  meeting  (kickoff) 

February  23,  1995 

Interim  results  distributed 

March  10,  1995 

Meeting,  submit  and  discuss  major  points 

March  23,  1995 

Meeting,  agree  on  major  report  points 

April  20,  1995 

Draft  report  distributed  for  Task  Force  review 

May  15,  1995 

Meeting  (final),  discuss  draft  report 

May  25,  1995 

Final  report  completed 

September  21, 1995 

1.3  TASK  FORCE  MEMBERSfflP 

The  membership  of  the  MSBTF  was  drawn  from  the  Services,  Office  of  the 
Secretary  of  Defense  (OSD),  Federally  Funded  Research  and  Development  Centers 
(FFRDCs),  and  private  industry.  LtCol  David  A.  Bartlett,  DMSO,  chaired  the  Task 
Force.  Composition  of  the  Task  Force  is  shown  in  Table  2. 


Table  2.  M&S  Benefits  Task  Force  Members 


Member 

Organization  Represented 

Mr.  Matt  Aylward 

MITRE  Corporation 

Ms.  Michelle  Bailey 

Naval  Air  Warfare  Center,  Weapons  Division 

LtCol  David  A.  Bartlett, 
USMC 

Defense  Modeling  and  Simulation  Office 

Mr.  John  Bishop 

Institute  for  Simulation  and  Training,  University 
of  Central  Florida 

Colonel  Guy  M.  Bourn,  USA 

Department  of  the  Army,  Office  of  the  Deputy 
Chief  of  Staff  for  Operations 

Mr.  Tom  Brown 

Arnold  Engineering  Development  Center 

Mr.  Albert  R.  Burge 

Office  of  the  Under  Secretary  of  Defense, 
Acquisition  and  Technology 

Mr.  Bob  Curtis 

DATAMAT  Systems  Research 

Mr.  Michael  L.  Darby 

North  Star  Institute 

Major  Dean  E.  Fish,  USMC 

Marine  Corps  Modeling  and  Simulation 

Management  Office 

Mr.  Trevor  Huth 

Loral  Federal  Systems 

Mr.  Don  Johnson 

Office  of  the  Under  Secretary  of  Defense, 

Personnel  and  Readiness 

Dr.  Franklin  L.  Moses 

Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences 

Mr.  Fred  A.  Meyers 

Office  of  the  Under  Secretary  of  Defense, 
Acquisition  and  Technology 

Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 

Mr.  Ron  Prishivalko 

Defense  Information  Systems  Agency 

Dr.  Henry  K.  Simpson 

Defense  Manpower  Data  Center 

Mr.  J.  Stephen  Schroeder 

Arnold  Engineering  Development  Center 

Mr.  Ernie  Smart 

Institute  for  Simulation  and  Training,  University 
of  Central  Florida 

Dr.  Robert  D.  Smith 

Naval  Air  Warfare  Center,  Weapons  Division 

Dr.  D.  Robert  Worley 

Institute  for  Defense  Analyses 

1.4  REPORT  OVERVIEW 

Quantified  impacts  for  M&S  applications  in  acquisition,  training,  and  analysis  are 
summarized  in  Chapters  2, 3,  and  4,  respectively. 

Appendices  A  and  B  survey  the  use  of  various  quantitative  and  qualitative 
measures  of  effectiveness  (MOEs).  Appendix  B,  in  particular,  describes  methods  of 
analyses  used  in  three  case  studies. 

A  list  of  references  is  provided  along  with  an  annotated  bibliography,  covering  the 
full  range  of  material  collected  by  the  Task  Force.  Abbreviations  and  acronyms  are  given 
at  the  end  of  the  report. 


2.  ACQUISITION  APPLICATIONS  OF  M&S 


2.1  TARGET  INTERACTION,  LETHALITY,  AND  VULNERABILITY  (TILV) 

TILV  refers  to  the  mechanisms  by  which  a  warhead  or  similar  device  can  defeat  a 
target  [TILV  1995].  The  TILV  area  addresses  the  tools,  methods,  databases,  and 
supporting  techniques  needed  to  assess  the  lethality  and  vulnerability  of  all  weapon 
systems,  including  aspects  of  design,  effectiveness,  and  survivability.  Modeling  and 
simulation  provide  a  significant  portion  of  the  TILV  capability  and,  in  particular,  an 
attractive  alternative  to  destructive  test.  Table  3  [TDLV  1995]  depicts  the  Return  on 
Investment  (ROI)  of  M&S  support  for  milestone  decisions  and  the  Cost  and  Operational 
Effectiveness  Analysis  process.  The  typical  ROI  was  between  $20  and  $30  returned  for 
each  $1  invested. 


Table  3.  TILV  Return  on  Investment 


Program 

Type  Analysis 

Total 

Invest 

($M) 

Direct 

Savings 

($M) 

R 

O 

I 

Program 

Result 

AMRAAM 

End  Game 

6.5 

250.0 

38 

Continued 

Bomb  Fragment  Data 

Arena  Tests 

0.0825 

0.9 

11 

Continued 

BLU-109 

Lethality  Testing 

0.0825 

3.0 

36 

Continued 

Air-to-Air  Missile 

Lethality  plus  Engagement 

20.0 

75.0 

4 

Continued 

Wide  Area  Anti- 
Armor  Munition 

Lethality  Analysis 

0.75 

30.0 

40 

Canceled 

Hypervelocity  Missile 

Lethality  Analysis 

0.5 

10.0 

20 

Canceled 

ISAS 

Lethality  Analysis 

0.75 

40.0 

53 

Canceled 

Kinetic  Energy 
Penetrator  (KEP) 

Lethality  Analysis 

1.1 

50.0 

45 

Canceled 

JP  233  Runway 

Attack  Munition 

Lethality  and  Vulnerability 
Analysis 

1.1 

54.0 

49 

Canceled 

Boosted  Kinetic 

Energy  Penetrator 

Runway  Vulnerability  Models 

2.75 

130.0 

47 

Canceled 

JAVELIN  ATOM 

Analytic  Simulation 

0.62 

14.0 

23 

M2  Bradley  FVS 

Engineering  Design 

0.88 

30.0 

34 

Msmmsmm 

Damage  Prediction 

1.83 

30.0 

16 

Cost  Avoidance 

MlA2Block3 

1.76 

100.0 

57 

Terminated 

Standard  Missile 

SM-2  BLKBIA 

Cost  Reduction 

2.25 

47.0 

21 

Accepted 

Phalanx  CIWS 

Performance  Evaluation 

8.12 

125.0 

15 

Continued 

Phalanx  CIWS 

Product  Upgrade 

6.63 

200.0 

30 

iiiiuiisissn 

Lethality  Analysis,  End  Game 

0.7 

16.0 

23 

Phoenix  Missile 

Lethality  Analysis,  End  Game 

2.23 

70.0 

31 

ECM  vs.  AMRAAM 

Lethality  Analysis,  End  Game 

0.58 

10.5 

18 

Eval.  Continues 

2.2  TECOM  VIRTUAL  PROVING  GROUNDS  (VPG) 

For  M&S  to  be  useful  and  valid,  the  applicable  tools  must  be  based  on  real  data 
derived  from  testing  [TECOM  1995].  The  Army’s  Test  and  Evaluation  Command 
(TECOM)  supports  this  concept  with  an  approach  known  as  Simulation  and  Modeling 
Anchored  by  Real  Testing  (SMART).  TECOM  is  employing  SMART  as  a  means  for 
researchers  and  developers  to  verify  that  their  models  and  simulations  are  based  on 
empirical  data.  VPG  is  a  network  of  models  and  simulations,  using  empirical  data,  that 
enables  interactive  testing  in  a  synthetic  environment.  A  number  of  projects  undertaken 
by  TECOM  use  these  models  and  simulations  to  determine  the  various  effects  on 
systems  and  to  replicate  actions  without  undertaking  the  time  and  expense  of  actual 
testing. 


Table  4  represents  a  summary  of  selected  TECOM  systems  that  used  VPG  in 
conducting  tests  and  evaluations  with  cost  avoidance  as  an  MOE.  Actual  cost  includes 
investment  in  simulation  when  appropriate  and  available. 

Table  4.  TECOM  VPG  Costs 


Project 

Use 

Simulation 

Actual 

Cost 

($M) 

Cost 

Avoidance 

(SM) 

Firing  Impulse 
Simulator 

Recoil  loads  and 
ballistic  shock  effects 

Replicate  actual  firing  without  the 
use  of  ammunition  for  tanks  and 
howitzers 

6.9 

23.0 

M830E1  Fuse 

Testing 

Evaluates  tank  vs. 

helicopter 

engagements 

Virtual  test  range  simulation 
using  simulated  helicopter 
engagements  with  manned  tank 

0.26 

1.5 

Moving  Target 
Simulator 

Immersion  of  entire 
weapon  system  (air 
or  ground)  into 
moving  visual  target 
environment 

Assess  the  ability  of  an  Ml  A2 
tank  crew  to  track  and  simulate 
firing  on  images  of  simulated 
maneuvering  targets 

— 

1.5 

per  year 

Simulation/Test 
Acceptance  Facility 
(STAF) 

Test  millimeter 
wave  radar-guided 
missiles 

Hardware-in-the-loop  simulator 
providing  test  of  a  “live”  missile 
with  multiple  computer-based  test 
scenarios 

— 

10.6 
per  year 

Aerial  Cable  Range 
(ACR) 

Test  missile 
tracking  of  heat 
sources 

Uses  a  3-mile  long  suspended 
Kevlar  cable  that  serves  as  path  for 
captive  vehicles 

0.7 

13.8 

Test  Item  Stimulators 
(TIS) 

Non-radiating 
simulated  message 
traffic  to  C3  systems 

Test  of  Enhanced  Position 

Location  Reporting  System 
(EPLRS) 

■ 

2.0 

Trajectory  Sense  and 
Destroy  Armor 
Simulation 
fSADARM) 

Model  ballistic 
simulation  for  the 
SADARM  projectile 

Enables  downrange  auto-trackers 
to  acquire  and  track  incoming 
projectiles  and  transition  quickly 
to  acquire  end-game  data 

— 

12.0 

HMlimHill 

Bridge  durability 
tests 

Mix  of  physical  and  simulated 
bridge  crossings 

0.325 

0.11 

2.3  ARMY  MISSILE  SYSTEMS 


The  Army  Missile  Command  (USAMICOM)  Research,  Development  and 
Engineering  Center  (RDEC)  uses  M&S  extensively  in  the  development  of  Army  Missile 
Systems  [Jolly  and  Ward  1995].  During  the  course  of  numerous  simulation  projects,  the 
benefits  of  hardware-in-the-loop  (HWIL)  simulations  have  translated  into  cost  savings 


and  avoidance  for  many  weapon  system  development  programs.  Examples  of  cost  saving 
and  avoidance  totaling  in  excess  of  $320M  are  presented  in  Table  5. 

Table  5.  Army  Missile  Systems  Costs 


Project 

Application  of  HWIL  and  DIS  Simulation 

Save/Avoid  ($M) 

MLRS-TGSM 

45%  reduction  in  flight/drop  test  program 

6.0 

FOG-M/NLOS 

HWIL  simulation  identified  all  hardware  and  software  faults 
prior  to  flight  tests,  resulting  in  reduction  in  flight  test 
costs 

15.0 

Longbow 

Successful  Proof-of-Principle  and  EMD  flight  test  programs 
with  prevention  of  at  least  2  test  failures  and  reduction  of 
risk  in  several  other  cases 

6.5 

Classified  Program 

Viability  of  this  development  program  possible  only 
through  HWIL  simulation;  estimated  flight  test  cost 
savings 

60.0 

HAWK 

Flight  test  cost  savings  on  counter  ECM  and  other  system 
improvements 

80.0 

STINGER 

Flight  test  cost  savings  for  benign,  countermeasured,  and 
untestable  scenarios 

>  90.0 

ATACMS 

Analysis  of  flight  test  anomaly  possible  only  with 
hardware-in-the-loop  simulation;  rapid  identification  of 
source  of  anomaly  saved  extensive  investigation 

0.5 

JAVELIN 

Performance  assessment  data  for  milestone  3  decision 
produced  by  simulations,  avoiding  several  flight  tests 

5.0 

Foreign  Materiel 
Exploitation 

ECM  hardware/software/techniques  evaluation  and 
optimization  against  foreign  threat  missiles  (Desert  Storm 
payoff  in  identified  saving  of  at  least  one  aircraft  and  pilot) 

>  25.0 

FAADS-BSFV  (DIS) 

Evaluate  options  using  real  soldiers,  without  requiring 
costly  development  of  prototype  systems,  and  save 
substantially  on  field  testing 

32.1 

Total  Cost  Savings  or  Avoidance 

>  320.1 

2.4  APACHE  LONGBOW 


An  example  of  some  of  the  benefits  of  using  M&S  in  the  Apache  Longbow 
program  are  summarized  in  Table  6  [Swinsick  1995].  Phase  I  of  Force  Development  Test 
and  Experimentation  (FDT&E)  was  based  on  manned  simulation.  Phase  II  employed 
approximately  the  same  test  scenario  and  activities  but  used  live  equipment.  Twice  as 
many  trials  were  conducted  in  Phase  I  than  in  Phase  II,  at  less  cost,  with  fewer  personnel, 
in  less  time.  Phase  I  tests  allowed  the  helicopter  crews  to  train  on  the  new  equipment 
without  the  risk  associated  with  flying  real  equipment.  It  also  allowed  development  and 
practice  of  new  tactics,  techniques,  and  procedures.  Those  responsible  for  developing 
scenarios  for  Initial  Operational  Test  and  Evaluation  (lOT&E)  have  the  opportunity  to 
structure  the  very  expensive  operational  test  to  gain  the  most  critical  information. 


Table  6.  Apache  Longbow  FDT&E  Tests 


Resources 

Phase  I 

Manned  Simulation 

Phase  II 

Field  Test 

$.712M 

$4.049M 

Equipment 

1  Simulator 

4AH-64D 

2  UH-60 

14  Ml  Tanks 

10  M3  Fighting  Vehicles 

2  2S6 

20  +  Air  Defense  Units 

47  +  Vehicles 

27 

663 

Mission  Tum-Around  Time 

2  Hours 

6  Hours 

Data  Reduction  Time 

4  Hours 

80  Hours 

Number  of  Trials 

32 

16 

Test  Period 

4  Weeks 

6  Weeks 

Safety 

No  Risk 

Moderate  Risk 

8 


3.  TRAINING  APPLICATIONS  OF  M&S 


3.1  INDIVIDUAL  SKILLS  TRAINING 

Individual  training  is  supported  most  often  by  stand-alone  simulators.  These 
simulators  range  from  simple  devices  (such  as  rifle  marksmanship  trainers)  to  more 
complex  devices  (such  as  maintenance  simulators,  tank  gunnery  simulators,  and  flight 
simulators).  Simpson  et  al.  [1995]  drew  these  general  conclusions  about  the  effectiveness 
and  cost  of  such  simulators; 

...in  aggregate,  simulators  provide  significant  beneficial  transfer  from 
simulator  to  aircraft  at  a  median  operating  cost  of  about  one-tenth  of  an 
aircraft... Because  of  their  scope,  the  body  of  studies  probably  provides 
the  strongest  case  for  the  value  of  any  type  of  simulation. 

Students  trained  using  maintenance  simulators  perform  about  as  well  as 
those  trained  with  actual  equipment,  but  simulators  cost  a  fraction. .  .of  the 
equipment... where  time  to  train  was  reported,  training  with  simulators 
took... less  time  than  with  actual  equipment. 

3.1.1  Aviation 

The  Army  estimates  that  it  substitutes  simulation  for  $68M  of  flight  operations 
training  in  the  active  force  and  $55M  in  the  Reserves  each  year.  The  Navy  considers 
simulation  to  be  effective  in  initial  training  in  unfamiliar  aircraft,  as  is  reflected  in  the  ratio 
of  simulator  to  actual  aircraft  training  flights  (40  to  77)  in  the  fleet  replacement  training 
program  for  F/A-18  aircraft.  The  Air  Force  Air  Mobility  Command  plans  to  replace  up 
to  50%  of  flight  training  hours  with  flight  simulators  and  other  training  devices  for  training 
air  transport  crews  [Orlansky  et  al.  1994;  Department  of  the  Army  1993]. 

The  operating  cost  of  flight  simulators  is  estimated  to  be  between  5  to  20%  that  of 
aircraft.  Many  studies  have  shown  that  skills  learned  in  flight  simulators  can  be 
performed  successfully  in  aircraft,  and  the  use  of  simulators  for  training  can  reduce  flight 
time  [Orlansky  and  String  1977].  In  a  more  recent  study,  the  median  cost  ratio  of 
operating  simulators  to  aircraft  was  estimated  to  be  8%  [Orlansky  et  al.  1984]. 


A  review  of  several  studies  showed  that  the  operating  costs  of  flight  simulators 
are  about  10%  of  actual  equipment  per  hour  trained  or  33%  if  acquisition  cost  is  taken 
into  account.  The  majority  of  tasks  trained  on  simulators  (59%)  have  significant  positive 
transfer  to  flight  performance  [Angier  et  al.  1993]. 

Bombing  and  air  drop  accuracy  data  indicate  that  additional  simulator  hours  seem 
to  have  a  greater  positive  effect  than  additional  flying  hours,  and  simulator  hours  cost  at 
most  a  third  as  much.  Helicopter  accident  data  indicate  that  both  flying  hours  and 
simulator  hours  reduce  accidents,  but  simulator  hours  do  not  increase  exposure  to  risk 
[Horowitz  et  al.  1992]. 

3.1.2  Small  Arms 

Several  studies  relating  to  the  use  of  simulation  in  lieu  of  live  fire  indicate  that 
performance  with  simulation  is  at  least  equal  to  live  fire  training,  but  that  cost  is  lower. 
Soldiers  with  MACS  (Multipurpose  Arcade  Combat  Simulator)  training  expended  less 
roimds  during  live-fire  qualifications  and  fewer  soldiers  failed  to  qualify  as  compared  to 
those  trained  using  traditional  methods.  Several  studies  with  the  Squad  Engagement 
Training  System  (SETS)  have  shown  positive  transfer  fi’om  SETS  to  live  fire.  Training 
with  the  Indoor  Simulated  Marksmanship  Trainer  (ISMT)  has  been  demonstrated  to 
benefit  live-fire  performance.  The  Precision  Gunnery  Training  System  (PGTS),  an 
inexpensive  trainer  for  TOW  and  DRAGON  missiles,  whose  rounds  are  very  expensive 
($11,500  and  $19,145,  respectively,  per  round),  has  been  demonstrated  to  be  cost 
effective,  and  also  permits  training  that  would  otherwise  cost  several  hundred  million 
dollars  per  year  if  actual  missiles  were  used  [Bailey  and  Hodak  1994;  Wilhoite  1993; 
Eisley  et  al.  1990;  Schendel  et  al.  1984;  Berg  et  al.  1993b]. 

3.1.3  Maintenance 

A  review  of  maintenance  simulators  found  that  they  are  as  effective  for  training  as 
actual  equipment  trainers  when  measured  by  student  achievement  in  school.  In  the 
majority  of  cases  examined,  the  cost  to  develop  and  fabricate  one  unit  was  less  than  60% 
of  actual  equipment  and  the  cost  of  fabricating  a  second  unit  was  less  than  20%. 
Acquisition  and  use  of  a  maintenance  simulator  over  a  15-year  period  cost  38%  as  much 
as  the  actual  equipment.  In  studies  where  time  to  train  was  reported,  simulators  took  25 
to  50%  less  time  than  actual  equipment  [Orlansky  and  String  1981]. 


3.2  COLLECTIVE  SKILLS  TRAINING 


Collective  training  focuses  on  tasks  performed  collectively  by  groups  of 
individuals  (e.g.,  crews,  teams,  units)  who  must  work  together  and  coordinate  their 
activities.  The  size  of  a  collective  may  vary  greatly  and  hence  collective  training  varies 
considerably  in  scale.  It  is  supported  most  commonly  by  live  or  virtual  simulation. 
Some  stand-alone  simulators  train  smaller  persormel  collectives  (e.g.,  flight  crews,  tank 
crews).  Advanced  distributed  simulation — a  set  of  varied  models  or  simulations  operating 
in  a  common  synthetic  environment  composed  of  live,  virtual,  and/or  constructive 
simulations — can  also  be  used  for  collective  training.  A  recent  analysis  by  the  General 
Accounting  Office  (GAO)  [GAO  1993]  conducted  for  Congress  cited  as  exemplary 
several  simulations  used  by  the  Army  for  collective  training:  COFT  (Conduct  of  Fire 
Trainer),  used  on  tanks  and  Bradley  Fighting  Vehicles;  MILES  (Multiple  Integrated  Laser 
Engagement  System),  used  to  simulate  direct  fire  weapons  fi-om  rifles  to  tank  and 
helicopter  gurmery  systems;  and  SIMNET  (simulator  networking),  used  to  provide  crew-, 
platoon-,  and  company-level  training.  A  1994  review  of  technologies  supporting  virtual 
simulation  indicated  that  it  is  becoming  increasingly  powerful  and  cost  effective  [OTA 
1994]. 

3.2.1  Crew/Team 

Evaluations  of  the  UCOFT  (Unit  Conduct  of  Fire  Trainer)  have  been  positive. 
Tank  guimers  trained  with  UCOFT  fire  their  opening  rounds  about  25%  faster  than 
conventionally  trained  gunners.  Based  on  an  analysis  of  a  hypothetical  force-on-force 
engagement,  UCOFT-trained  gunners  would  be  expected  to  kill  significantly  more 
opposing  tanks  than  conventionally  trained  gunners  [Operational  Research  and  Analysis 
Establishment  1990].  Boldovici  et  al.  [1985]  reviewed  UCOFT  tests  and  concluded  that 
UCOFT  provides  improvements  in  gunner  proficiency.  Substantial  gains  were  found  in 
percents  of  targets  acquired,  engaged,  hit,  and  killed  for  groups  undergoing  sustainment 
and  transition  training.  Gains  were  attributed  to  improvements  in  acquisition  time, 
engagement  time,  and  first-round  hits,  which  in  turn  allowed  time  to  scan,  acquire,  and 
engage  available  second  and  third  targets.  Hughes  et  al.  [1988]  evaluated  the  training 
effectiveness  of  the  UCOFT  empirically  with  369  tank  commander-gunner  pairs  and 
found  that  UCOFT  training  accelerated  skill  acquisition,  improved  performance  in 
subsequent  training  events,  and  was  well  accepted  by  users. 

In  tank  gunnery,  the  introduction  of  COFT  reduced  the  annual  expenditure  of 
ammunition  from  134  to  100  rounds  per  tank  and  improved  marksmanship.  This  resulted 
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in  an  annual  cost  avoidance  of  approximately  $29M.  The  new  Tank  Weapons  Gunnery 
Simulation  System  is  expected  to  reduce  the  annual  consumption  to  78  rounds,  for  an 
additional  saving  of  $21 M  to  $50M  each  year  [Orlansky  et  al.  1994;  Department  of  the 
Army  1993;  Morrison  et  al.  1991a,  1991b;  Tumage  and  Bliss  1990]. 

3.2.2  Multiship  Air  Combat 

In  evaluations  of  developmental  distributed  interactive  simulation  (DIS)  systems 
designed  to  support  multiship  air  combat  training  in  a  combat  engagement  simulation 
environment,  participating  pilots  and  air  weapons  controllers  indicated  that  simulation 
enhanced  their  combat  readiness  and  was  more  beneficial  in  some  areas  than  traditional 
imit  training  [Bell  and  Crane  1992;  Houck  et  al.  1991]. 

In  evaluations  of  a  SIMNET-compatible  air  combat  simulator,  pilots  received 
training  and  then  rated  their  interest  in  receiving  additional  training  on  each  of  30  tasks. 
Tasks  with  the  highest  rated  interest  can  usually  be  practiced  only  in  large  exercises  or 
cannot  be  practiced  except  in  simulation.  It  was  concluded  that  multiplayer  simulator- 
based  training  is  a  valuable  training  medium  for  increasing  wartime  readiness,  especially 
for  less  experienced  pilots  [Crane  and  Berger  1993]. 

3.2.3  Tactical  Ground  Combat 

During  the  Persian  Gulf  War,  at  the  battle  of  “73  Easting,”  U.S.  troops  destroyed 
an  opposing  force  three  times  their  size  while  fighting  in  an  area  the  Iraqis  had  previously 
used  for  training  exercises.  Leaders  of  the  U.S.  force  cited  the  training  they  had  received 
with  live  simulation,  virtual  simulation,  and  stand-alone  crew  training  simulators  as 
important  factors  in  their  success  [Orlansky  1993]. 

The  Army  Science  Board  [1989]  has  estimated  that  simulators  would  enable  a 
reduction  in  aviation  and  vehicle  OPTEMPO  (Operating  Tempo)  and  training  ammunition 
by  15  to  20%  while  maintaining  the  same  or  better  level  of  unit  performance. 

A  series  of  tests  and  evaluations  have  demonstrated  SIMNET’s  value  for 
collective  training.  Schwab  and  Gound  [1986]  evaluated  SIMNET’s  capability  to 
support  platoon-level  command  and  control  exercises  to  train  individual  and  collective 
tasks.  The  eight  platoons  were  divided  into  two  groups,  one  with  prior  SIMNET  training 
and  the  other  without.  Three  of  the  four  platoons  in  each  group  improved  their 
performance  between  the  first  and  second  set  of  situational  training  exercises.  The 
SIMNET  group  improved  its  average  group  score  by  13%  while  the  baseline  group 


improved  its  score  by  6%.  Findings  of  Kraemer  and  Bessemer  [1987]  suggest  that 
SIMNET  training  helped  units  develop  and  improve  their  fire  control  distribution  plans 
and  helped  unit  leaders  develop  the  command,  control,  and  communications  skills  to 
effectively  execute  those  plans  during  platoon  battle  runs.  Brown  et  al.  [1988]  found  that 
SIMNET  training  increased  field  exercise  platoon  performance,  command  and  control,  and 
leadership  skills,  and  adequately  portrayed  vehicle  and  battlefield  sounds.  SIMNET  also 
improved  performance  of  command  and  control,  platoon  movement,  leadership,  and  fire 
distribution  during  the  company  team  Army  Training  and  Evaluation  Program  (ARTEP). 
Burnside  [1990]  found  that  35%  of  ARTEP  Mission  Training  Plan  (MTP)  tasks  can  be 
trained  with  SIMNET.  Bessemer  [1991]  found  positive  transfer  of  tactical  training  from 
SIMNET  to  field  training.  Analysis  of  an  effectiveness  comparison  between  SIMNET 
and  home-station  field  training  indicates  that  SIMNET  is  extremely  effective  in  increasing 
performance  for  SIMNET-trainable  tasks  relative  to  field  training.  Tradeoff  analyses 
show  that  investment  in  SIMNET-like  facilities  could  be  repaid  by  an  8  to  14%  decrease 
in  OPTEMPO  [Angier  et  al.  1993]. 

An  analysis  of  the  training  capabilities  and  cost  effectiveness  of  the  Close  Combat 
Tactical  Trainer  (CCTT)  concluded  that  it  has  the  potential  to  train  tasks  relating  to 
command,  control,  and  communications;  maneuver  and  navigation;  and  teamwork  and 
leadership.  When  fielded,  CCTT  would  be  cost  effective  and  its  life  cycle  costs  would  be 
paid  back  fully  diuing  its  service  life  [Noble  and  Johnson  1991]. 

3.2.4  Multi-Service  and  Joint  Training 

The  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  has 
successfully  demonstrated  the  use  of  virtual  simulation  for  multi-Service  close  air  support 
training  and  is  currently  expanding  its  demonstration  platform  to  include  the  Joint  fire 
support  mission  [ARI  1995a,  1995b;  Hawley  and  Christ,  in  press]. 

Virtual  simulation  has  the  potential  to  enable  Joint  and  inter-Service  training  in 
mission  areas  not  being  trained  sufficiently  now  (e.g.,  close  air  support).  The  technology 
permits  coordinated  training  among  the  Services  while  individual  Service  elements  remain 
at  their  home  stations  [Simpson  et  al.  1995]. 

3.3  COMMAND  AND  STAFF  TRAINING 

Command  and  staff  training  occurs  within  constructive,  live,  and/or  virtual 
simulations.  The  participating  commanders  and  staffs  range  fi’om  the  lowest  to  the 
highest  echelon  and  fi'om  a  single  Service  up  through  Multi-Service,  Joint,  and  Combined 


commands.  The  most  economical  way  to  conduct  such  training  is  with  constructive 
simulations,  as  they  enable  commanders  and  staffs  to  experiment  without  the  cost  of  fuel, 
ammunition,  and  military  personnel.  Command  and  staff  training  does  occur  during  live 
and  virtual  simulations,  but  usually  these  simulations  are  intended  to  train  all  participants 
at  all  levels.  Because  of  their  economy  and  relative  ease  of  implementation,  constructive 
simulations  have  proliferated  in  many  different  training  domains. 

3.3.1  Single-Service  Training 

The  1990  REFORGER  (Return  of  Forces  to  Germany)  exercise  made  extensive 
use  of  constructive  simulation  to  train  leaders  at  brigade,  division,  and  corps  level. 
Benefits  of  such  training  were  the  emphasis  on  battle  planning,  staff  procedures,  and 
command  and  control;  more  efficient  use  of  training  time;  focus  on  higher  echelons  that 
would  otherwise  be  cost  prohibitive;  and  reduced  adverse  environmental  and  political 
impacts.  The  transportation  and  cargo  handling  costs  of  the  1990  exercise  were  more  than 
$4M  less  than  costs  historically  [GAO  1991].  In  1992,  constructive  simulation  was  used 
to  avoid  $34M  in  costs  as  compared  with  the  equivalent  exercise  done  without  simulation 
in  1988.  Participants  also  believed  that  the  training  of  staffs  and  planners  involved  was 
improved  [Simpson  et  al.  1995].  However,  previous  REFORGER  exercises  satisfied  a 
treaty  obligation  to  return  forces  to  Germany,  and  they  provided  extensive  training  to 
those  responsible  for  physical  movement  of  troops  and  equipment. 

GAO  noted  that  at  the  brigade  level  and  above,  simulations  can  be  used  to 
improve  the  decision-making  skills  of  senior  battle  officers  before  they  command  units  in 
large-scale  training  exercises  [GAO  1993]. 

Formal  evaluations  have  demonstrated  that  constructive  simulations  train 
commanders  and  staffs  effectively  and  are  relatively  inexpensive.  The  JANUS(A)  is 
effective  in  training  company  level  officers  and  platoon  leaders  on  current  tactics  and 
doctrine.  The  Brigade/Battalion  Battle  Simulation  (BBS)  has  proven  effective  at  training 
brigade  and  battalion  staffs  [Bryant  et  al.  1992]. 

3.3.2  Multi-Service  and  Joint  Training 

The  Defense  Science  Board  [1988]  concluded  that  computer-based  simulated 
scenarios  offer  the  only  practical  and  affordable  means  to  improve  the  training  of  Service 
operational  commanders,  their  staffs,  and  the  commanders  and  staffs  who  report  to  them. 
Battle  simulation  offers  the  only  opportunity  to  practice  the  use  of  certain  weapon 
systems,  sensors,  tactics,  and  techniques  against  a  skilled  adversary. 


Agile  Provider  (AP),  a  Joint  exercise  sponsored  by  the  United  States  Atlantic 
Command  (USACOM),  replaced  Unified  Endeavor  (UE)  in  1995.  AP  was  a  field  exercise 
last  held  in  1994.  UE  was  supported  by  a  Joint  Training  Confederation  (JTC)  of  models 
interacting  through  the  Aggregate  Level  Simulation  Protocol  (ALSP).  The  models 
replaced  steaming  days  and  fl)dng  hours,  and  focused  on  the  primary  training  audience, 
the  JTF  commander  and  staff  Total  costs  for  AP-94  were  $40M,  with  $8M  in  strategic 
lift  costs.  UE-95’s  costs  totaled  $2.9M  with  approximately  $0.5M  in  strategic  lift. 
Approximately  85%  of  the  UE-95  participants  rated  their  training  as  good  and  82%  rated 
it  better  than  a  similar  field  exercise  like  AP-94.  The  conclusion  was  better  training  at 
7.5%  of  the  cost. 
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4.  ANALYSIS  APPLICATIONS  OF  M&S 


M&S  contributes  to  innumerable  decisions  involving  system  evaluation  and  force 
sizing.  In  addition,  it  contributes  significantly  to  combat  operations.  In  1990  and  1991, 
the  Air  Force  Studies  and  Analyses  Agency  (AFSAA)  performed  a  series  of  Gulf  War 
analyses  that  Lieutenant  General  Glosson  (then  chief  of  Central  Air  Forces  Special 
Projects)  asserted  “...saved  literally  hundreds  of  lives”  [Case  1991]. 

A  team  of  AFSAA  analysts  was  quickly  deployed  to  the  Air  Force  Operations 
Center  in  Riyadh,  Saudi  Arabia,  where  they  analyzed  the  air  campaign  both  before  and 
after  it  began.  For  their  analyses,  they  used  primarily  the  Army’s  Space  and  Strategic 
Defense  Command’s  Extended  Air  Defense  Simulation  (EADSIM)  (also  called  the 
C3ISIM)  model. 

At  the  time,  EADSIM  was  a  new  model  to  AFSAA  and  had  been  selected  because 
it  did  an  excellent  job  of  analyzing  command,  control,  and  commumcations.  It  was  a 
hybrid  model  with  Monte  Carlo  and  deterministic  features.  The  combat  operations 
planners  were  able  to  watch  a  preview  of  the  attack  as  it  unfolded  in  a  way  that 
graphically  revealed  the  plan’s  strengths  and  weaknesses.  Unlike  the  actual  air  defenses, 
the  modeled  air  defenses  acted  in  a  rational  manner,  with  the  simulation  results  showing  a 
worst  case  scenario  for  the  actual  air  assault. 

One  main  contribution  was  to  choreograph  the  masses  of  aircraft  into  and  out  of 
the  Kuwaiti  Theater  of  Operations.  To  ensure  that  aerial  tankers  would  make  their 
rendezvous  with  fighters  in  need  of  refueling,  missions  were  played  out  in  advance. 
Attacks  were  carefully  choreographed  to  avoid  mid-air  collisions,  especially  during  the 
first  day’s  intense  activity. 

Planners  also  analyzed  the  best  use  of  defense  suppression  assets,  and  alerted 
planners  of  missions  that  were  too  hazardous  for  some  aircraft.  For  instance,  the  AFSAA 
team  analyses  indicated  that  it  would  be  too  dangerous  to  carry  out  plans  to  send  A-6  and 
Tornado  aircraft  directly  over  Baghdad.  As  a  result,  only  F-117  stealth  fighters  (none  of 
which  was  lost)  were  assigned  targets  in  this  highly  defended  area.  Undoubtedly,  these 
changes  saved  allied  lives  and  the  needless  loss  of  aircraft.  When  planners  determined  that 
Scud  sites  in  Western  Iraq  were  too  well  defended  and  (as  existing  prior  to  the  attack)  too 
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hazardous  for  F-15E  attacks,  defense  suppression  missions  were  reconfigured  to  correct 
the  problem.  When  aircraft  losses  did  occur,  computer  simulations  were  used  to  help 
determine  the  most  likely  cause  so  that  later  missions  could  be  made  less  dangerous. 


APPENDIX  A.  MEASURES  OF  EFFECTIVENESS 


A.1  INTRODUCTION 
A.1.1  Problem  Statement 

DMSO  has  articulated  a  need  for  assessing  the  effect  of  M&S  on  the  full  range  of 
DoD  activities.  The  effect  is  difficult  to  quantify  for  several  reasons:  acceptable 
effectiveness  metrics  are  lacking,  supporting  data  are  difficult  to  unearth,  and  in  some 
cases  it  is  not  possible  to  identify  a  baseline  from  which  to  measure. 

One  member  of  the  Task  Force'  examined  30  programs  to  determine  how  users  of 
M&S  measured  their  success.  Those  programs  are  listed  and  summarized  in  the 
Attachment  after  the  exposition  of  findings.  A  second  member  of  the  Task  Force 
presented  a  complementary  discussion,  contained  in  Appendix  B,  showing  alternative 
methods  of  calculating  cost  effectiveness  and  a  range  of  effectiveness  measures  and  cost 
components. 

To  measure  the  effect  of  the  many  M&S  applications  in  the  DoD,  we  must  first 
state  our  objectives  in  quantifiable  terms.  Only  then  can  we  assess  our  progress  toward 
reaching  those  objectives.  The  metrics  described  here  are  nominated  for  comment,  and 
will  move  us  toward  measuring  the  effects  of  M&S. 

A.  1.2  Technical  Areas,  Functional  Areas,  and  DoD  M&S  Objectives 

M&S  capabilities  fall  into  one  of  two  broad  areas.  The  Technical  Areas  deals  with 
mechanisms  that  make  the  M&S  application  work,  while  the  Functional  Areas  considers 
how  the  M&S  application  will  be  used.  The  two  broad  areas,  their  subareas,  and  the 
DoD  M&S  objectives  are  discussed  in  the  following  list,  along  with  a  short  description  of 
the  M&S  objectives. 

•  Technical  Areas.  As  mentioned  above,  this  area  deals  with  the  inner  workings  of  the 

M&S  tool.  The  particular  topics  in  this  area  are  as  follow: 

Architecture:  The  high-level  system  and  software  design  of  the  M&S  tool. 

Computer  Generated  Forces  (CGFV  The  representation  of  constructive  simulations. 


*  Mr.  Matt  Aylward  of  MI  TRE,  Quantico,  VA. 
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Environmental  Representation:  How  the  real  world  is  portrayed  in  the  synthetic 
environment;  the  effects  of  weather,  terrain,  obscurants,  and  their  interaction  with  the 
exercise  entities. 

Information/Databasei  Methods  for  M&S  tools  to  store  or  access  information,  data 
modeling. 

Interoperability:  How  various  M&S  tools  interface  and  operate  together. 

Networking:  How  information  is  shared  among  physically  remote  M&S  tools. 

Verification.  Validation  &  Accreditation  (W&A):  the  process  of  giving  M&S  tools  an 
official  stamp  of  approval. 

Instrumentation:  Details  of  the  infi'astructure  needed  to  incorporate  live  entities  into 
the  synthetic  environment;  the  hardware  and  software  that  allows  real  personnel  and 
platforms  to  send  their  state  variables  between  M&S  tools,  typically  through  electronic 
messages. 

•  Functional  Areas.  M&S  tools  are  used  in  the  following  ways: 

Analysis:  To  conduct  experiments  where  useful  information  can  be  extracted. 

Training:  To  enhance  military  readiness  through  training,  mission  planning,  and 
mission  rehearsal. 

Acquisition  (research  and  development!:  To  allow  virtual  prototyping,  enhance 
brainstorming,  and  expand  the  number  of  design  options  that  can  be  considered. 

Acquisition  (test  and  evaluation!:  To  accomplish  both  Developmental  Testing  (DT) 
and  Operational  Testing  (OT).  Breadboards  and  brassboards,  combined  with 
stimulation  from  M&S  tools,  enhance  the  quality  of  DT  and  OT  results. 

Acquisition  (production  and  logistics V  To  support  design,  manufacturing,  process 
analysis,  and  support  planning. 

•  Master  Plan  Objectives. 

The  Department  of  Defense  Modeling  and  Simulation  Master  Plan  [DMSO  1995] 
lists  objectives  that  M&S  tools  should  achieve.  The  relevant  objectives — ^framework, 
environmental  representation,  systems  representation,  and  human  behavior 
representation — are  closely  related  to  the  preceding  sections  on  Technical  and 
Functional  Areas.  Accordingly,  a  separate  discussion  of  each  M&S  application  area  is 
unnecessary. 
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A.2  OBSERVATIONS  FROM  30  CASE  STUDIES 


A.2.1  The  Search  for  Metrics 

To  capture  the  benefits  of  any  investment,  there  must  be  agreement  upon  objective 
standards  to  measure  the  performance  of  a  particular  investment.  This  appendix  contains 
a  proposed  set,  with  quantitative  and  qualitative  metrics.  The  projects  listed  in  this 
appendix  were  selected  to  demonstrate  the  breadth  of  M&S  benefits  enjoyed  by  DoD.  A 
discussion  of  the  candidate  quantitative  and  qualitative  metrics  follows. 

A.2.2  Quantitative  Metrics:  Technical  Areas 

•  Architecture:  Decisions  about  programming  architecture  have  far  reaching 
consequences.  Not  only  are  current  and  future  M&S  tools  affected,  legacy  systems 
will  also  feel  some  effect.  The  metric  in  this  area.  Percent  of  Legacy  Migration, 
reflects  this  effect.  An  underlying  Measure  of  Performance  (MOP)  to  this  metric  may 
be  the  amount  of  effort  (measured  in  staff-years  or  dollars)  required  to  migrate  a 
legacy  system  to  the  proposed  architecture. 

•  Computer  Generated  Forces  (CGF):  Modular  Semi-Automated  Forces  (ModSAF)  is 
the  current  DIS-compatible  software  for  generating  personnel-  and  vehicle-level 
entities.  M&S  tools  featuring  highly  detailed  constructive  simulation  of  combat  at  the 
lowest  level  rely  on  this  software.  War  games  also  use  CGF  but  usually  at  a  higher 
level  of  aggregation.  Following  the  same  rationale  developed  for  Architecture,  the 
candidate  metric  is  Percentage  of  Software  Reused. 

•  Environmental  Representation:  The  candidate  metric.  Stimulations,  reflects  the  role  of 
the  environment  in  the  real  world.  Military  personnel  constantly  adjust  their  plans 
according  to  the  current  or  future  environment  conditions,  sorties  are  canceled  or 
diverted  due  to  obscurants  over  the  primary  target,  or  offensive  operations  are  delayed 
because  the  main  supply  route  is  a  sea  of  mud.  This  metric  can  be  measured  in  many 
ways.  For  example,  how  many  environmental  effects  can  the  M&S  application 
portray?  How  many  variables  does  the  application  use  to  describe  the  water  column 
near  the  landing  beach?  So,  within  the  metric  of  Stimulation,  a  number  of  MOPs  can 
be  developed  for  a  given  M&S  application. 

•  Human-Systems  Interface:  There  are  quantitatively  rigorous  techniques  for  measuring 
how  well  a  M&S  application  approximates  the  real  system;  accordingly,  the  candidate 
metric  is  User  Acceptance. 
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•  Information/Database:  Ready  access  to  information,  especially  information  stored  in 
databases,  is  critical  to  reducing  the  overhead  of  M&S  tools.  As  future  tools  will  rely 
on  information  archived  in  databases  of  various  designs,  database  design  decisions 
should  consider  migrating  legacy  databases.  Possible  metrics  include  Level  of  Effort 
Required  for  Collaboration  and  Reuse,  measured  in  staff-years  and  cost  avoidance, 
perhaps  arising  from  the  reuse  of  an  existing  database. 

•  Networking:  The  links  connecting  the  various  sites  involved  in  a  DIS  exercise  are 
vital  for  exercise  success.  When  choosing  among  various  network  designs,  the 
analysis  should  identify  the  least  expensive  choice  that  ensures  a  successful  DIS 
exercise.  Therefore,  possible  metrics  are  Cost  per  Megabit  per  second  and  Latency. 

•  Verification.  Validation,  and  Accreditation  fW&AV  If  there  is  a  suitable  M&S 
application  in  existence,  and  this  application  has  undergone  a  formal  W&A  process, 
then  a  particular  project  has  no  reason  to  create  a  new  application.  Developing  a 
rigorous  W&A  procedure,  particularly  one  that  is  not  onerous  to  the  owners  of  M&S 
tools,  would  enhance  software  reuse;  accordingly,  a  candidate  metric  is  Cost 
Avoidance,  arising  from  such  reuse. 

•  Instrumentation:  When  conducting  instrumented  exercises,  the  Services  can 
completely  forgo  live  fire  or  use  the  position  reporting  capability  to  have  positive 
location  information  on  each  player.  In  either  case,  the  risk  of  fratricide  or  erroneous 
fire  mission  approval  by  the  Fire  Support  Coordination  Center  is  greatly  diminished. 
Consequently,  a  candidate  metric  is  Risk  Reduction. 

A.2.3  Quantitative  Metrics:  Functional  Areas 

•  Analysis:  Current  evaluations  of  military  operations  rely  heavily  on  anecdotal 
information;  if  analysts  had  a  mechanism  to  capture  what  really  happened,  the  resulting 
conclusions  could  have  a  much  smaller  confidence  interval.  An  assessment  of  how 
various  proposed  M&S  tools  allow  the  analyst  to  better  understand  a  process,  and 
perhaps  conduct  the  analysis  from  “ground  truth,”  would  be  of  obvious  interest.  One 
candidate  metric  for  this  assessment  is  Net  Utility. 

•  Training:  By  far  the  functional  area  to  benefit  the  most  from  M&S  tools,  training  also 
has  the  most  easily  defined  metrics.  Examples  include  Cost  Savings,  Cost  Avoidance, 
and  Risk  Reduction,  for  example,  it  is  hard  to  have  a  fatal  mishap  in  an  F/A-18 
simulator. 


•  Acquisition  ^research  and  development):  Candidate  metrics  include  Cost  Savings,  due 
to  avoidance  of  premature  fabrication,  and  Number  of  Options  Considered.  The 
second  metric  for  research  and  development  (R&D)  and  production  and  logistics 
(P&L)  captures  the  same  idea:  Study  a  wide  range  of  options  for  the  proposed  system 
in  virtual  reality  before  making  the  first  bend  in  metal. 

•  Acquisition  (test  and  evaluation):  A  number  of  case  studies  show  that  using  M&S 
tools  to  prepare  for  DT  and  OT  is  advantageous.  M&S  allows  for  better  designed 
tests,  aids  in  training  the  test  force,  and  identifies  areas  of  deficient  data  collection.  In 
some  cases,  such  as  testing  software  upgrades  for  the  F-14,  M&S  is  the  only  way  to 
conduct  DT,  due  to  the  risks  involved.  Use  of  Cost  Avoidance  as  a  metric  is  well 
supported  by  these  case  studies. 

•  Acquisition  (production  and  logistics):  Candidate  metrics  include  Cost  Savings  and 
Number  of  Options  Considered.  The  former  accrues  when  production  lines  and 
consumption  rates  can  be  simulated,  allowing  problem  identification  and  correction 
before  the  system  is  fielded.  The  latter  is  appropriate  when  application  of  M&S  allows 
planners  to  explore  a  wide  range  of  production  options  in  detail,  at  a  minimal  cost, 
when  compared  to  exploring  the  options  with  real-world  equipment. 

A.2.4  Qualitative  Metrics:  Technical  Areas 

•  Computer  Generated  Forces  fCGFI:  The  more  realistic  the  portrayal  by  the  CGF,  the 
better  the  training;  therefore,  a  candidate  metric  is  Training  Quality,  as  evaluated  by 
the  training  audience. 

•  Environmental  Representation:  One  candidate  metric  is  Immersion,  a  measure  of  how 
“real”  the  environment  feels  to  the  trainee. 

•  Human-Svstems  Interface:  How  well  the  M&S  application  matches  the  feel  of  the 
simulated  system  is  critical  to  positive  training  transfer.  For  this  reason,  a  candidate 
metric  is  Ease  of  Use,  as  evaluated  by  the  trainee  or  instructor. 

•  Interoperability:  Ideally,  all  Service  models  would  interoperate  readily,  thus  ensuring 
that  the  conclusions  of  a  study  sponsored  by  one  Service  would  be  acceptable  to  all 
Services.  For  example,  the  Office  of  the  Secretary  of  Defense  (OSD)  can  choose 
among  a  number  of  combat  simulations  to  evaluate  Service  roles  and  missions;  the 
selected  simulation  should  neither  over-  nor  understate  the  relative  value  of  any  one 
Service.  A  candidate  metric.  Level  Playing  Field,  embodies  this  idea  of  a  neutral 
evaluation  tool. 
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•  Networking:  If  the  M«&S  application  is  distributed,  the  network  can  either  degrade  or 
enhance  the  sensory  experience  of  the  participants.  The  quality  of  sensory  stimulation, 
the  feeling  of  Immersion  experience  by  players,  is  a  candidate  metric. 

•  Verification.  Validation,  and  Accreditation  (W&A):  The  candidate  metric  is 
Enhanced  Decision  Support.  Here,  the  implicit  assumption  is  extended  one  step 
further:  A  model  that  has  completed  a  formal  W&A  process  produces  results  that 
are  “more  valid”  than  results  from  a  non- W&A  process.  “More  valid”  results  will 
lead  to  better  quality  decisions. 

•  Instrumentation:  With  many  M&S  tools  executing  over  current  command,  control, 
communications,  computer,  and  intelligence  (C4I)  systems,  it  is  desirable  for  these  two 
disciplines  to  merge.  A  candidate  metric  is  Merger  of  C41  with  M&S.  This  metric 
estimates  how  closely  a  particular  M&S  application  approaches  this  goal. 

A.2.5  Qualitative  Metrics:  Functional  Areas 

•  Analysis:  As  discussed  in  the  quantitative  section,  much  analysis  of  military  operations 
is  based  on  anecdotal  data.  Merging  live,  virtual,  and  constructive  simulations  with 
C4I  systems  would  give  analysts  much  better  quality  data  with  which  to  work.  For 
these  reasons  the  metric.  Data  Quality,  is  offered. 

•  Training:  The  candidate  metrics  are  Readiness  and  Unique  Training.  Readiness  is 
enhanced  because  the  lower  cost  of  training  with  M&S  tools  increases  the  quantity  and 
quality  of  training  opportunities.  Unique  Training  is  possible  with  M&S  because  only 
electrons  are  in  danger  of  getting  killed.  This  metric  speaks  to  the  ability  of  M&S  to 
present  trainees  with  situations  not  seen  in  the  real  world  outside  of  combat. 

•  Acquisition  (research  and  development!:  The  candidate  metrics  are  Brainstorming 
and  Unique  Capability.  Brainstorming  refers  to  the  ability  of  M&S  to  let  program 
managers  explore  a  much  wider  array  of  options  before  settling  on  one  approach.  In 
other  cases,  the  Unique  Capability  of  M&S  cannot  be  reproduced  in  the  real  world. 

•  Acquisition  ftest  and  evaluation^:  Possible  metrics  include  Developmental  Test 
Planning,  Operational  Test  Planning,  Development  of  Measures  of  Performance 
(MOP),  and  Development  of  Measures  of  Effectiveness  (MOE).  These  metrics  assess 
how  well  an  M&S  tool  assists  the  test  and  evaluation  community  in  all  of  the 
foregoing  tasks. 
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Projects  Examined  for  Measures  of  Effectiveness 

1.  Indoor  Simulated  Marksmanship  Trainer  (ISMT):  A  stand-alone  M&S  application, 
ISMT  is  used  for  training  Marines  in  all  aspects  of  small  arms  fire,  as  well  as  training 
small  units  in  tactical  engagements.  Analysis  indicates  that  use  of  the  ISMT  for  a 
portion  of  annual  weapons  requalification  could  save  significant  amounts  of  money  as 
a  result  of  ammunition  offset  [Fish  1995a]. 

2.  Deployable  Forward  Observer  -  Modular  Universal  Laser  Equipment  (DFO-MULE): 
A  stand-alone  M&S  application,  DFO-MULE  is  a  training  device  for  forward 
observers  (artillery  and  mortar)  and  forward  air  controllers.  It  complies  with  current 
DIS  standards.  The  DFO-MULE  is  being  used  in  the  Multi-Service  Distributed 
Testbed.  A  MITRE  analysis  indicated  that  use  of  the  DFO-MULE  for  required 
forward  observer  training  could  save  significant  amounts  of  money  as  a  result  of 
ammunition  offset  [Fish  1995c].  Assuming  a  10%  offset  in  live  fire  tasks,  savings  in 
ammunition  expenditures  could  recover  the  acquisition  costs  before  the  end  of  the 
second  year. 

3.  Emerald  Light:  A  Marine  Corps  proof-of-concept  demonstration.  Emerald  Light  will 
instrument  a  training  range  at  the  Marine  Corps  Air  Ground  Combat  Center 
(MCAGCC)  at  29  Palms,  CA.  Ultimately,  the  MCAGCC  and  the  National  Training 
Center  (NTC)  at  Fort  Irwin,  CA,  will  be  linked.  This  will  allow  the  conduct  of  Joint 
exercises  at  both  sites.  During  the  exercise,  participants  will  share  a  synthetic 
battlespace  over  the  Defense  Simulation  Internet  (DSI). 

4.  Synthetic  Theater  of  War  -  Europe  (STOW-E):  A  Defense  Advanced  Research 
Projects  Agency  (DARPA)  funded  effort,  STOW-E  allowed  the  Army  to  link  live, 
virtual,  and  constructive  simulations  to  conduct  a  large-scale  training  event  embedded 
within  ATLANTIC  RESOLVE.  Comparable  in  breadth  to  REFORGER  (Return  of 
Forces  to  Germany),  ATLANTIC  RESOLVE  cost  on  the  order  of  $37  million  less  to 
conduct  than  REFORGER. 
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5.  LeatherNet:  A  DARPA-funded  project  developed  in  concert  with  STOW-97, 
LeatherNet  seeks  to  create  a  credible  Marine  Corps  CGF  at  the  level  of  the  individual 

rifleman. 

6.  Turret  Layout:  Not  really  a  project  as  much  as  a  study,  the  Turret  Layout  effort 
compared  the  use  of  M&S  to  prototype  construction  for  developing  modifications  to 
the  Abrams  tank.  The  M&S  application  allowed  more  options  to  be  considered,  at  a 
lower  cost,  in  less  time,  while  involving  the  user  community;  the  benefits  were  clear 

cut  and  convincing. 

7.  Advanced  Medium  Range  Air  to  Air  Missile  (AMRAAM):  During  the  development 
of  this  system,  investments  in  M&S  tools  allowed  engineers  to  fly  a  complete  mission 
profile  in  a  virtual  environment.  The  amount  and  quality  of  data  made  available  from 
this  investment  far  exceeded  the  telemetry  from  a  real  flight.  Also,  given  the  cost  of 
each  flight  and  the  number  of  flights  required,  actual  flight  testing  was  prohibitively 
expensive.  Only  M&S  could  satisfy  the  need  for  performance  data  at  an  acceptable 

cost. 

8.  AIM-9X;  The  AIM-9X  missile  is  an  implementation  of  advanced  medium  range  air- 
to-air  missile  technology.  See  discussion  of  the  AMRAAM  (7). 

9.  F-14  Software  Test:  The  F-14,  like  most  modem  aircraft,  relies  on  computers  to 
execute  its  mission.  Changes  to  its  software  are  made  continuously,  yet  each  change 
could  potentially  result  in  non-desirable  flight  performance  such  as  crashes.  For  this 
reason,  each  change  in  software  must  be  rigorously  tested  before  the  plane  is  flown. 
This  testing  must  be  conducted  on  the  ground,  with  all  flight  control  systems 

receiving  accurate  input  stimulation.  Only  M&S  tools  can  provide  this  input.  In  the 

absence  of  simulation,  upgrades  to  the  F-14  software  would  not  be  possible. 

10.  Advanced  Amphibious  Assault  Vehicle  (AAAV):  While  this  program  is  moving 
forward  under  the  conventional  acquisition  paradigm,  it  is  concurrently  looking  at 
ways  to  use  M&S  tools  to  change  the  process.  One  example  is  the  participation  of  the 
two  automotive  test  rigs  participating  in  the  Virtual  Proving  Ground.  In  this  project, 
data  collected  from  real  vehicles  on  the  test  track  at  Aberdeen  Proving  Grounds  are 
compared  to  data  produced  by  computer  simulations.  Successful  completion  of  the 
Virtual  Proving  Ground  project  will  give  designers  the  ability  to  consider  many 
design  options  without  bending  any  metal. 
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11.  Boeing  111:  During  the  development  of  this  aircraft,  the  Boeing  Corporation  re¬ 
invented  itself:  this  is  the  world’s  first  paper-less  airplane.  Boeing’s  corporate 
information  system  architecture  allowed  for  extensive  use  of  CAD,  CAE,  and  CAM 
(computer-aided  design,  engineering,  and  manufacturing).  Production  of  the  aircraft 
was  greatly  simplified,  as  only  a  minuscule  amount  of  time  was  lost  to  poorly  fitting 
parts.  In  turn,  this  reduced  the  costs  of  production  by  reducing  labor  overtime  charges 
and  scrap  rework. 

12.  New  SSN  Prototype:  Employing  simulation  (see  Boeing  777,  {11))  for  the  follow-on 
to  the  SSN-21  (Sea  Wolf  class).  General  Dynamics  has  already  experienced  cost 
avoidance  on  the  order  of  tens  of  millions  of  dollars.  Proposed  changes  to  the  weapon 
systems  consoles,  sensor  suites,  or  engineering  plant  can  be  eompletely  explored  in  a 
virtual  environment  before  any  metal  is  bent.  Compared  to  previous  construction 
methods,  large  costs  are  avoided.  This  is  an  example  of  how  the  corporate 
information  system  architecture  can  have  a  significant  effect  on  the  bottom  line. 

13.  B-2  Mockup:  In  a  vein  similar  to  Boeing  and  General  Dynamics,  Northrop  designed 
its  CISA  to  gather  information  throughout  the  company.  As  a  result,  the  mockup  of 
the  B-2  was  so  close  to  the  design  resident  in  the  CISA  that  future  mockups  may  be 
eliminated  entirely. 

14.  SEEK  IGLOO:  An  Air  Force  project  to  deploy  warning  radar,  SEEK  IGLOO’ s 
concept  of  employment  called  for  manned  installations.  A  MITRE  simulation 
determined  that  the  radar  was  much  more  reliable  than  assumed.  This  led  to  a 
different  concept  of  employment,  unmanned  radar  installations  of  smaller  size.  Large 
savings  from  cost  avoidance  were  realized. 

15.  F-16  Operational  Test  (OT)  Scenario  Development:  By  using  simulators,  the  OT 
project  officer  was  able  to  realize  a  number  of  benefits.  First,  the  test  team  was  fully 
trained  in  the  scenario  for  the  OT,  increasing  the  efficiency  of  the  test.  Second,  the 
test  crews  were  able  to  show  the  project  officer  what  performance  measures  were 
truly  important,  leading  to  a  modification  of  the  test  scenario.  In  this  way,  simulation 
led  to  a  higher  quality  OT. 

16.  Forward  Area  Air  Defense  System/Air  Defense,  Anti-Tank  System  (FAADS/ 
ADATS)  Measure  of  Performance  (MOP):  Since  neither  of  these  systems  exist,  the 
project  officer  was  stymied  in  attempts  to  develop  appropriate  MOPs.  The  use  of 
simulation  allowed  the  project  officer  to  clarify  the  concept  of  employment  and 
develop  worthwhile  MOPs. 


17.  Non  Line  of  Sight  (NLOS)  Operational  Test;  The  NLOS  is  a  new  type  of  anti-tank 
weapon  that  allows  for  precision  attack  of  armored  vehicles  by  a  gunner  in  full 
defilade.  Facing  problems  similar  to  the  FAADS/ADATS  (7(5),  the  project  officer 
turned  to  simulation.  Again,  the  existence  of  sophisticated  M&S  tools  led  to  high 
quality  OT  of  a  future  system. 

18.  Virtual  Proving  Ground:  This  is  an  effort  between  the  Army’s  Aberdeen  Proving 
Grounds,  Aberdeen,  MD,  and  the  University  of  Iowa.  It  is  an  attempt  to  create  a 
synthetic  environment  for  testing  vehicles.  The  goal  is  to  allow  engineers  to  fully 
explore  system  design  (e.g.,  of  the  HMMWV)  before  any  metal  is  bent. 

19.  Joint  Warfare  Concept  Analysis  -  Operations  Research  (JWCA-OR):  JWCA-OR  is  an 
effort  to  improve  the  quality  of  Joint  analysis.  For  this  work,  it  is  essential  that  all 
Services  are  represented,  as  JWCA-OR  supports  force  structure  decisions,  aids  in 
developing  Joint  doctrine,  and  guides  force  allocation  to  the  warfighting  commands. 
Currently,  Service  representation  largely  involves  legacy  systems. 

20.  Joint  Warrior  Interoperability  Demonstration  (JWID):  JWID  is  a  series  of 
demonstrations  sponsored  by  the  Chairman,  Joint  Chiefs  of  Staff.  They  are  primarily 
concerned  with  C4I;  future  demonstrations  will  see  further  integration  of  M&S  with 
C4I. 

21.  Standard  Interchange  Format  (SIF):  Developed  by  the  Institute  for  Simulation  and 
Training  (1ST),  Orlando,  FL,  SF  allows  existing  databases  to  interface  with  M&S 
tools.  The  use  of  SF  generates  savings  for  each  project  by  reducing  the  number  of 
years  normally  required  to  develop  a  custom  interface. 

22.  B-52  Data  Study;  Undertaken  by  the  Strategic  Air  Command  (SAC)  during  Operation 
Desert  Shield/Desert  Storm,  this  effort  collected  a  wide  array  of  operational  data  from 
the  bomber  force.  This  data  is  potentially  very  useful  for  a  number  of  different  M&S 
functional  uses. 


23.  Jedi  Knights:  A  colloquialism  that  refers  to  a  group  of  Army  officers  who  provided 
operations  research  support  to  the  Commander  in  Chief,  U.S.  Central  Command 
(CINCCENT),  in  theater.  Drawn  from  the  Command  and  General  Staff  College,  the 
members  of  this  group  were  experts  in  the  TACWAR  model.  Prior  to  the  start  of 
Operation  Desert  Storm,  the  Jedi  Knights  ran  TACWAR  manually  and  compared  the 
results  to  the  computer  results  from  the  same  scenario.  They  judged  the  TACWAR 
output  credible  and  proceeded  to  use  TACWAR  for  operational  support.  The  Jedi 
Knights  are  an  example  of  the  effort  and  benefits  associated  with  YV&A  of 
simulations. 

24.  Desert  Storm  Operations  Research:  CINCCENT,  as  well  as  subordinate  commanders, 
made  extensive  use  of  operations  research  personnel  as  plans  for  Operation  Desert 
Storm  were  developed  or  executed.  Analytic  support  was  provided  from  the  United 
States,  as  well  as  from  Operation  Research  (OR)  cells  in  theater.  Plans  were 
developed,  analyzed,  and  modified  in  a  greatly  truncated  cycle.  Without  sophisticated 
M&S  tools,  the  OR  cells  would  have  been  unable  to  respond  to  the  needs  of  the 
operational  commanders.  The  benefit  of  simulation  was  especially  evident  in  planning 
and  conducting  the  air  campaign. 

25.  Joint  Surveillance  Target  Acquisition  Radar  Terminal  Emulation  (JSTAR  TE): 
Originally  intended  as  an  adjunct  to  the  JSTAR  program,  the  JSTAR  TE  allowed  the 
JSTAR  to  reach  operational  capability  in  time  for  Desert  Storm,  six  years  ahead  of 
schedule.  This  was  a  great  success,  both  for  the  war  effort  as  well  as  for  the  program. 

26.  Defense  Information  Systems  Agency  (DISA):  DISA  is  responsible  for  developing  the 
communication  links  necessary  to  connect  the  far  flung  activities  of  the  DoD.  In  this 
task,  DISA  has  used  a  number  of  M&S  tools  to  consider  various  alternative  methods 
of  linking  the  DoD  nodes.  There  are  several  documented  instances  of  cost  avoidance 
that  are  the  direct  result  of  using  M&S. 

27.  Simulation  Utility  Management  System  (SUMS):  SUMS  is  an  Air  Force  effort  to 
develop  an  M&S  tool  to  assess  the  effects  of  changing  manpower  policies  and 
programs.  It  also  allows  personnel  planners  to  consider  various  scenarios  regarding 
the  nature  of  the  ci\dlian  labor  pool. 
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28.  Virtual  Medicine:  This  project  is  still  in  the  basic  research  phase,  but  it  offers 
tantalizing  benefits.  Battlefield  surgeons  could  operate  without  subjecting  the 
wounded  to  the  trauma  of  transportation  to  a  field  hospital.  This  multiplies  the 
effectiveness  of  each  surgeon,  while  reducing  demands  on  the  transportation  system 
and  eliminating  a  lucrative  rear  area  target,  the  large  field  hospital. 


APPENDIX  B.  METHODS  OF  ANALYSIS 


One  member  of  the  Task  Force  gathered  examples  of  data  and  cost-effectiveness 
analysis  with  which  she  was  familiar.^  Through  three  case  studies,  she  depicted  alternative 
methods  of  calculating  cost  effectiveness  using  the  same  data.  Subsequently,  she 
#  generalized  her  observations  and  discussed  a  range  of  effectiveness  measures  and  cost 

components,  contained  in  this  appendix.  The  sections  covered  in  this  appendix  are  as 
follows: 


B.  1  Alternative  Calculations  for  Three  Case  Studies 
B .  1 . 1  AMRAAM  Hardware  in  the  Loop 
B.1.2  F/A-18  Weapons  Software  Support  Facility  (WSSF) 
B.1.3  Kernel  Blitz 

B.2  Is  Cost  Savings  the  Best  Measure  of  M&S  Effectiveness? 
B.3  Candidate  Measures  of  Effectiveness 
B.4  Identification  of  Expenditures 


^  Ms.  Michelle  Bailey  of  Naval  Air  Warfare  Center,  Weapons  Division  (NAWCWD),  China  Lake,  CA. 
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B.l  ALTERNATIVE  CALCULATIONS  FOR  THREE  CASE  STUDIES 

The  separate  case  studies  were  chosen  because  of  the  availability  of  data  to  the 
author  and  their  ability  to  demonstrate  the  various  methods  of  analysis.  The  data  come 
from,  or  have  been  examined  by,  personnel  associated  with  the  relevant  programs.  Four 
different  methods  of  calculating  cost  effectiveness  are  shoAvn,  each  providing  a  different 
result. 

•  The  first  and  most  common  method  calculates  cost  savings  or  avoidance,  and 
is  usually  based  on  the  assumption  that  live  and  simulated  events  are 
completely  interchangeable. 

•  The  second  method  is  break-even  analysis  that  determines  how  many  live 
events  must  be  replaced  by  simulated  events  to  recover  capital  investment  and 
operating  costs  in  a  given  period  of  time. 

•  The  third  method  is  based  on  the  assumption  that  finding  errors  early  in  the 
acquisition  process  is  less  costly  to  repair  than  finding  them  later  in  the 
process. 

•  The  final  method  of  comparing  alternative  events  is  to  compare  their  costs  and 
their  effectiveness  separately.  While  it  is  the  most  general  method,  it  is  also 
difficult  to  implement  due  to  the  inability  to  adequately  measure  the 
effectiveness  of  an  event.  This  final  method  would  allow  comparison  of  an 
alternative  M&S-supported  event  Avith  a  baseline  event  where  the  alternative 
was  more  costly  but  provided  better  training,  for  example.  Military  experts 
could  then  decide  whether  the  training  increment  or  decrement  was  worth  the 
additional,  or  conversely,  lower  cost. 

B.1.1  AMRAAM  Hardware  in  the  Loop 

The  AMRAAM  Hardware-in-the-Loop  (HIL)  facility  at  Point  Mugu,  CA,  is 
employed  in  the  ongoing  evaluation  of  missile  guidance  and  control  system  performance. 
The  facility  includes  a  flight  simulator  table,  anechoic  chamber,  target  simulators,  special 
interface  hardware,  and  an  instrumented  missile.  The  facility  can  be  used  for  additional 
applications,  but  only  its  use  for  testing  the  AMRAAM  is  considered  here.  Its  primary 
cost  components  are  shown  in  Table  B-1. 


Table  B-1.  AMRAAM  HIL  Costs 


BRAC  Replacement  Cost^ 

$23.7  M 

Yearly  Operating  Cost 

$930K  (SIM) 

Number  of  Tests  per  Year 

8,400 

Using  the  assumption  that  all  firings  are  live,  we  calculate  an  extremely  favorable 
cost  savings,  as  shown  in  Table  B-2.  While  8,400  simulated  firings  per  year  are  possible, 
a  program  simply  could  not  afford  8,400  live  firings.  Even  so,  several  M&S  cost  savings 
we  have  gathered  have  been  calculated  using  this  type  of  assumption. 


Table  B-2.  AMRAAM  Example  Savings  Calculations 


Cost  Savings  Method  (assumes  all  live  firings) 

Cost  per  firing 

$40K 

Cost  of  missile 

+  $250K 

Total  cost  per  firing 

$290K 

Number  of  tests 

X  8.400 

Total  savings 

$2,43  6M 

Could  a  program  do  8,400  firings,  let  alone  in  one  year? 

For  this  case,  break-even  analysis  may  be  more  meaningful.  An  example  is  shown 
in  Table  B-3.  In  this  case,  recovery  of  BRAC  replacement  costs  occurs  in  10  years, 
assuming  that  12  firings  per  year  are  simulated,  a  far  more  reasonable  assumption. 
Further,  assuming  3  or  4  firings  per  year  beyond  the  10-year  break-even  point  will  recoup 
facility  operations  and  maintenance  costs.  The  conclusion,  then,  is  that  12  firings  per  year 
will  recoup  the  capital  investment  in  10  years,  and  the  other  8,388  simulated  firings  are 
value  added,  i.e.,  contribute  more  effectiveness.  It  must  be  remembered  that  an 
AMRAAM  missile  costs  considerably  more  during  its  development  and  early  production. 
Consolidation  of  the  earlier  missile  costs  with  later  production  costs  would  shorten  the 
payback  time. 


^  Base  Realignment  and  Closure  (BRAC)  replacement  costs  represent  the  cost  to  replace  a  fecility  and 
not  the  cost  of  original  development  and  maintenance.  This  metric  is  used  here  because  it  is  a  certified 
figure  with  a  definite  meaning  applied  uniformly  across  the  country. 
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Table  B-3.  AMRAAM  Example  Break-Even  Calculations 


_ Time  to  Break-Even  Method _ 

•  Number  of  firings  required  for  break-even  in  10  years  at  12  firings  per  year 
(23.7  +  10  X  (operating  costs)/(cost  per  firing))  =  $34M  /  $290K 

=  117  firings  in  10  years 

•  Number  of  firings  saved  per  year  to  maintain  cost  effectiveness 

($1M  /  $290K  =  3.45  firings  per  year) _ _ _ 


A  third  alternative  method  of  calculating  the  cost  effectiveness  of  this  type  of 
facility  is  to  record  the  number  of  errors  found  during  HTL  testing  that  would  have  caused 
a  firing  failure.  Live  firings  are  an  expensive  way  of  finding  errors.  We  do  not  have  the 
data  required  to  conduct  this  type  of  analysis.  However,  we  can  approach  it  using  the 
F/A-18  WSSF  data,  discussed  in  the  next  section. 

B.1.2  F/A-18  Weapons  Software  Support  Facility  (WSSF) 

The  F/A-18  Weapons  Software  Support  Facility  (WSSF)  at  China  Lake,  CA,  is 
used  for  integration,  checkout,  and  verification  and  validation  (V&V)  of  avionics  software 
with  actual  avionics  hardware  operating  as  a  total  aircraft  system.  The  WSSF  is  actually 
several  facilities  containing  avionics  hardware,  simulations  of  flight  dynamics,  weapons 
simulations,  and  operator  consoles.  Table  B-4  shows  the  WSSF  cost  factors  used  in  the 
following  example  calculations. 


Table  B-4.  F/A-18  WSSF  Cost  Factors 


BRAC  Replacement  Cost 

$54  M 

Yearly  Operating  Cost 

$6  to  8  M 

Number  of  Test  Hours  per  Year 

over  6,000 

Lab  Costs  per  Hour  (F/A-18) 

$930 

Lab  Costs  per  Hour  (other  aircraft) 

$1550 

Ground  Costs  per  Hour 

$100 

Flight  Costs  per  Hour 

$2,800 

The  WSSF  is  also  used  by  weapons  programs  for  integration  and  checkout  of  their 
aircraft  interfaces.  In  addition,  it  has  been  used  to  supply  simulated  aircraft  for  other  tests. 
The  cost  savings  being  computed  (Tables  B-6  and  6)  are  just  the  savings  for  the  F/A-18, 
not  for  all  programs  using  the  facility.  It  is  important  to  note  that  the  cost  per  flight  used 
here  is  the  actual  figure  charged  to  the  project.  The  true  fully  amortized  cost  of  keeping 
an  F/A-18  in  the  air  and  flight  ready  is  probably  much  higher.  The  topic  here  is 
methodology. 


Table  B-5.  F/A-18  WSSF  A/B  Software  Upgrade 


Cost  Component 

Hours  Expended 

%  Errors  Found 

Lab  Hours 

1,084 

73% 

Ground  Hours 

81 

2% 

Flight  Hours 

195 

11% 

Other  Methods'* 

? 

14% 

Table  B-6. 

F/A-18  WSSF  C/D  Software  Upgrade 

Cost  Component 

Hours  Expended 

%  Errors  Found 

Lab  Hours 

4,957 

61% 

Ground  Hours 

440 

4% 

Flight  Hours 

966 

13% 

Other  Methods 

? 

22% 

The  more  errors  that  can  be  found  in  the  early  stages  of  development  using  WSSF, 
the  cheaper  the  overall  program  will  be  without  even  considering  safety  issues.  Ground 
tests  are  relatively  cheap,  but  they  can  only  be  used  for  simple  power  checks.  They  are 
included  here  so  that  total  errors  add  up  to  100%.  Table  B-7  depicts  example  savings 
calculations. 


Table  B-7.  WSSF  Example  Savings  Calculations 


Cost  Savings  Method  (assumes  use  of  flight  hours  for  all  lab  debug) 

Cost  of  flight  hours  (A/B) 

1,084  hrs  X  $2,800/hr 

$3M 

Cost  of  flight  hours  (C/D) 

4,957  hrs  x  $2,800/hr 

+  $14M 

Total  cost  of  flight  hours 

Annual  operating  costs 

Savings  per  year 

$10M 

However,  there  are  not  enough  local  planes  to  fly  6,000  hours  in  one  year. 

The  break-even  viewpoint,  based  on  replacement  and  operating  costs  (Table  B-8), 
yields  a  more  reasonable  number  of  flight  hours,  but  even  that  is  difficult  for  one  test 
facility  to  bear.  If  each  lab  hour  equated  to  a  flight  hour,  we  would  need  more  than  one 
facility  testing  the  software,  or  we  would  need  more  F/A-18s  dedicated  to  software 
integration  and  test.  The  WSSF  actually  has  several  labs  which  may  be  used  in  parallel. 


Includes  code  reviews  and  other  paper-based  checks. 
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Table  B-8.  WSSF  Example  Break-Even  Calculations 


Time  to  Break-Even  Method 

Replacement  costs 

$54M 

Maintenance  costs  ($7M  per  year) 

+  $70M 

Total  costs 

$124M 

Cost  per  flight 

$2,880 

Have  to  save  4,400  flights  per  year  for  10  years 

($124M/$2,880  =  4,428) 

The  real  value  added  of  the  WSSF  is  that  an  aircraft  as  complex  as  the  F/A-18  is 
not  possible  without  this  type  of  test  facility.  We  could  not  fly  enough  to  test  it.  There  is 
a  danger  in  just  looking  at  cost  savings  as  the  measure  of  whether  or  not  we  invest  in 
M&S.  As  we  demand  more  from  our  warfighting  systems — safer,  more  accurate,  more 
environmentally  friendly,  more  stealthy,  longer  range,  etc. — we  will  have  to  demand  more 
from  our  test  and  training  systems. 

B.1.3  Kernel  Blitz 

Kernel  Blitz  was  a  fleet  training  exercise  (FLEETEX)  that  included  live  ships, 
submarines,  aircraft,  and  land  troops.  The  simulation  portion  augmented  the  fleet  with 
additional  synthetic  ships,  submarines,  aircraft,  and  weapons.  The  simulation  center  used 
several  existing  computer  facilities  (including  both  coasts)  and  existing  communications 
capability  to  link  to  platforms.  A  purpose  of  the  exercise  was  to  show  that  the  use  of 
simulated  assets  could  add  realism  and  complexity  to  training  exercises.  The  costs  of 
these  simulated  assets  are  depicted  in  Table  B-9. 


Table  B-9.  Kernel  Blitz 


Cost  of  Assets  Simulated 

Ships  and  submarines 

(23  platforms  x  2  days  x  SlOOK/day) 

$4.6M 

Aircraft 

(27  platforms  x  4  hrs  x  $3K/hr) 

+$0.3M 

Weapons 

(23  weapons  x  $500K/weapon) 

+$11.5M 

Costs 

BFTT  Enhancements 

Total  Savings 

$16M 

The  Battle  Force  Tactical  Trainer  (BFTT)  existed  prior  to  Kernel  Blitz  but  was 
enhanced  for  this  exercise.  During  Kernel  Blitz,  33  real  ships  and  submarines  were  used. 
The  simulated  assets  significantly  increased  that  number.  The  commanders  at  sea  quickly 


forgot  who  was  real  and  who  was  not.  The  fleet  commands  will  have  to  answer  the 
question  of  whether  they  would  ever  put  55  ships  and  subs  into  a  training  exercise. 

The  $500K  per  weapon  may  seem  high  to  some,  but  it  is  the  value  used  by  the 
BFTT  office  (the  AMRAAM  is  running  about  $250K  per  copy).^  Regardless  of  what  we 
think  about  the  full  cost  of  all  the  simulated  assets,  the  $350K  modification  costs  are 
impressively  low  (they  represent  only  those  costs  charged  to  Kernel  Blitz  via  BFTT).  If 
we  counted  only  the  use  of  two  ships  for  two  days  ($400K),  the  Navy  would  recoup  its 
investment. 

A  study  by  the  Center  for  Naval  Analyses  (CNA)  [Neuberger  and  Shea  1995] 
states  that  “At  this  point,  simulation  should  be  viewed  as  enriching  training  and  increasing 
readiness  rather  than  reducing  costs.”  The  CNA  analysis  also  specified  much  greater 
costs.  However,  the  purpose  of  this  discussion  is  not  to  determine  the  cost  effectiveness 
of  Kernel  Blitz,  but  to  demonstrate  possible  analytic  methods. 


*  There  may  likely  be  different  cost  factors  included  in  the  $500K  and  $250K  values. 


B.2  IS  COST  SAVINGS  THE  BEST  MEASURE  OF  M&S  EFFECTIVENESS? 

There  are  four  basic  categories  of  effectiveness  measures  obtained  from  applying 
modeling  and  simulation— c/o/wg  it  better,  doing  it  faster,  doing  it  cheaper,  doing  it  at  all. 
By  “doing  it  better,”  we  mean  that  the  quality  of  the  product  or  the  quality  of  the 
processes  employed  is  improved  through  the  application  of  M&S.  This  is  sometimes  hard 
to  measure  in  terms  of  dollar  savings.  What  value  do  we  put  on  safer  processes?  We 
know  how  to  determine  the  cost  of  a  disaster,  but  what  about  the  cost  of  near  misses?  Is 
there  a  savings  from  reducing  the  number  of  near  misses?  We  can  usually  obtain  dollar 
savings  for  more  accurate  testing,  earlier  discovery  of  problems,  and  repeatability  of 
testing,  although  there  will  be  some  subjectivity  in  the  figures. 

Simulations  make  it  possible  to  conduct  training  events  or  test  events  that  would 
not  be  possible  or  affordable  if  conducted  live.  Is  it  reasonable  to  compare  the  cost  of  a 
simulated  event  to  the  cost  of  a  live  event  that  never  would  have  occurred?  Better  would 
be  to  compare  the  costs  and  effects  of  two  realistic  but  different  live  and  simulated  events. 
However,  we  often  lack  the  appropriate  effectiveness  measures. 

Sometimes,  it  just  would  not  be  possible  to  conduct  a  specific  test,  or  train  for  a 
specific  situation,  without  simulation.  For  instance,  in  testing  seekers,  there  are  neither 
enough  test  points  nor  space  to  hook  up  test  equipment  to  obtain  all  the  information 
needed  about  the  behavior  of  the  hardware.  By  using  a  simulation,  we  have  access  to  all 
parameters.  For  aircraft,  we  want  them  to  be  able  to  withstand  a  certain  amount  of  G- 
loading,  but  to  actually  test  that  would  mean  risking  the  loss  of  an  aircraft  at  the  edge  of 
its  envelope — so  we  simulate  the  effects  of  Gs  through  application  of  M&S. 

When  classifying  the  effectiveness  of  M&S,  much  depends  upon  its  application.  Is 
it  a  wargame  or  an  engineering  simulation?  Identifying  effectiveness  measures  for  the 
different  M&S  applications  would  make  it  easier  for  users  to  keep  track  of  the  effects  of 
M&S.  For  instance,  wargames  may  save  money  by  identifying  shortfalls  of  existing 
weapons  by  pointing  out  tactical  solutions  vice  acquisition  solutions,  or  by  identifying  a 
set  of  equally  effective  solutions  from  which  the  least  expensive  can  be  chosen. 
Engineering  simulations  save  money  by  enabling  faster  design.  But  what  is  the  value 
added  by  increasing  the  number  of  alternatives  considered  through  simulation? 


B.3  CANDIDATE  MEASURES  OF  EFFECTIVENESS 


Table  B-10  lists  candidate  measures  of  effectiveness  (MOEs)  and  how  they  may  be 
derived  or  calculated.  The  application  space  is  mission  planning. 

Table  B-10.  Mission  Pianning 


Candidate  MOE 

Determination  of  MOE 

Actual  time  savings  of  commanders 
doing  the  strategy,  per  mission 

Review  mission  planning  times 

Value  added  of  “quick  reaction” 
capability;  shorten  war,  avoid 
casualties 

Wargame  with  and  without  M&S 
capability 

Dolt 

Better 

Value  added  of  additional  (on-line) 
information  to  mission  planners 

Review  costs  of  mistakes, 
wargaming 

Value  added  of  considering 
additional  strategies 

Wargame  with  and  without  M&S 
capability 

Value  added  of  considering  multiple 
enemy  reactions  to  strategy 

Review  costs  of  unexpected 
reactions 

Dolt 

Cheaper 

Cost  savings  of  using  new  methods 

Review  cost  of  current  equipment 
and  compare  to  projected  costs 

Dolt 

At  All 

Value  added  of  retargeting  mission 
en  route 

Number  of  “wasted”  missions 

Value  added  of  more  detailed 
planning 

Operator  assessment  of  mistakes 
caused  by  ambiguity 

Value  added  of  automatic  recording 
or  of  review  of  strategies,  scenarios 
and  lessons  learned 

Evaluate  mission  planning  training 
methods 

Table  B-11  lists  candidate  MOEs  for  M&S  tools  that  support  analysis  in  support 
of  Research,  Development,  Test  and  Evaluation  (RDT&E).  Examples  of  this  type  of 
analysis  includes  cost  estimation,  technical  effectiveness  evaluation,  and  cost  and 
operational  effectiveness  analysis  (COEA). 
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Table  B-11.  RDT&E,  Analysis 


Candidate  MOE 

Determination  of  MOE 

Better  adherence  to  schedule 

Use  of  virtual  prototyping 

Review  daily  program  expenditures 

Look  at  turn-around  time  for 
physical  models 

Do  It 

Better 

Value  of  adding  more  detailed 
analysis 

Review  number  of  design,  software, 
planning  changes 

Value  added  of  considering  more 
alternatives 

Review  pre-planned  product 
improvement,  cost  reduction, 
packaging  efforts 

Value  added  of  making  better 
decisions 

Estimate  of  unknown  unknowns, 
number  of  backup  plans  used  for  risk 
mitigation 

Dolt 

Cheaper 

Cost  savings  of  using  new  methods 

Review  cost  of  current  equipment 
purchase,  use,  and  maintenance  and 
compare  to  projected  costs 

Dolt 

At  All 

Value  added  of  “executable 
requirements” 

Costs  of  erroneous  requirements: 
suits,  redesigns,  ambiguities 

Value  added  of  operators  of  virtual 
prototyping 

Costs  of  failing  operational 
evaluation 

Table  B-12  lists  candidate  MOEs  for  M&S  tools  employed  in  the  design  phase  of 
RDT&E.  Examples  include  trade-off  studies,  engineering  simulations,  parametric 
optimization,  maintenance  planning,  logistics  planning,  and  production  planning. 


Table  B-12.  RDT&E,  Design  Phase 


Candidate  MOE 

Determination  of  MOE 

Reduction  of  design  iterations 

Compare  to  similar  efforts 

Automatic  design  documentation 

Compare  to  manual  methods 

Dolt 

Incorporation  of  maintenance. 

Estimates  of  reduced  life  cycle  costs 

Better 

logistics,  and  production 
considerations 

from  what  simulations  pointed  out 

Use  of  virtual  prototyping 

Cost  of  physical  models 

Evaluation  of  designs  under  more 

Estimated  costs  of  design  failure 

At  All 

situations 

under  those  situations 
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Table  B-13  offers  seven  MOEs  for  M&S  tools  used  in  test  and  evaluation. 

Table  B-13.  Test  and  Evaluation 


Candidate  MOE 

Determination  of  MOE 

I^Q|||||||H 

Better  adherence  to  schedule 

Better  use  of  flight  test  time 

Daily  cost  of  ranges,  program  slips 

Percent  of  test  time  wasted 

Do  It 

Better 

Value  added  of  “monte  carlo-ing” 
test  conditions 

Percent  of  operational  requirements 
not  physically  tested  but  inferred 
from  testing 

Value  added  of  rehearsing  test 

Percent  of  tests  wasted 

Earlier  identification  of  problems 

Look  at  cost  and  spending  curves 
for  phase  of  project,  look  at  cost  of 
engineering  change  proposals  by 
phase 

Dolt 

Cheaper 

Use  of  virtual  prototyping 

Cost  of  physical  models 

Dolt 

At  All 

Evaluation  of  designs  under  more 
situations 

Estimated  costs  of  design  failure 
under  those  situations 

Table  B-14  lists  seven  candidate  MOEs  for  M&S  application  to  training  and  how 
those  MOEs  might  be  determined. 

Table  B-14.  Training 


Candidate  MOE 

Determination  of  MOE 

Cost  savings  for  fewer  training  days 

Value  added  of  training  en  route 

Review  average  number  of  days  for 
specific  training 

Percent  delay  in  deployment  due  to 
training 

Dolt 

Better 

Value  added  of  exposing  trainees  to 
more  situations 

Total  assessment  of  trainee  progress 

Review  of  operator  errors 

Evaluation  of  individualized  training 
to  graduate  some  individuals  early 

Dolt 

Cheaper 

Cost  savings  of  using  new  methods 

Review  cost  of  current  methods  and 
equipment 

Dolt 

At  All 

Individual  remedial  training 

Virtual  reality  training  in  hazardous 
situations 

Review  number  of  “flunked”  trainees 

Review  casualties,  accidents  due  to 
operator  error 
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And  finally,  Table  B-15  lists  candidate  MOEs  for  M&S  tools  in  support  of  military 
operations. 

Table  B-15.  Support  to  Military  Operations 


Candidate  MOE 

Determination  of  MOE 

Dolt 

Faster 

Logistics  routing 

Time  saved  with  better  method 

Do  It 

Better 

Weapons  mix  studies,  both  platform 
and  individual 

Enhancement  of  capabilities  from 
tailored  mixes 

Dolt 

Cheaper 

Reduction  of  personnel  required  to 
do  analysis 

Amount  of  analysis  based  upon 
simulation 

Dolt 

At  All 

Decision  aids 

Benefits  of  faster,  better  decisions 

B.4  roENTIFICATION  OF  EXPENDITURES 

The  previous  sections  identified  several  MOEs  for  M&S.  This  section  takes  a 
closer  look  at  the  costs.  Ideally,  cost  estimation  would  be  the  responsibility  of  each 
program  manager  when  determining  whether  to  pursue  M&S  versus  other  options.  Too 
often  we  examine  only  the  costs  of  building  the  simulation  (or  enhancing  an  existing  one) 
and  forget  about  the  cost  of  V&V,  training,  operation,  and  maintenance. 

An  additional  shortcoming  in  assessing  the  cost  effectiveness  of  M&S  is  that  the 
cost  of  failure  is  rarely  captured.  With  the  large  up-front  costs  associated  with  some 
M&S  tools,  a  considerable  amount  of  money  can  be  spent  before  determining  that  the 
tool  simply  will  not  work.  Unfortunately,  these  lessons  learned  rarely  get  publicized,  so 
we  “learn”  the  same  lessons  repeatedly. 

The  costs  of  an  M&S  tool  are  less  dependent  upon  its  application  domain  and 
more  dependent  upon  its  physical  implementation.  If  a  simulation  is  entirely  software,  its 
costs  can  be  identified  in  the  same  fashion  as  any  other  software  system.  The  same  is  true 
of  hardware-in-the-loop  and  live  simulations. 

The  greatest  difficulty  in  acquiring  data  is  getting  the  right  data  and  understanding 
its  meaning  and  limits.  It  is  imperative  that  the  M&S  community  decide  what  data  it  needs 
and  provide  guidelines  to  program  managers  on  how  to  record  that  data.  It  does  not  have 
to  be  difficult  if  people  know  fi’om  the  beginning  what  is  needed. 


needs  and  provide  guidelines  to  program  managers  on  how  to  record  that  data.  It  does  not 
have  to  be  difficult  if  people  know  from  the  beginning  what  is  needed. 

The  second  problem  is  making  sure  the  data  are  used  correctly.  Good  data  used 
against  the  provider  will  not  engender  more  good  data.  This  is  a  political  problem  and 
hence  more  difficult  to  solve  than  the  first. 

Table  B-16  summarizes  the  identification  of  expenditures. 

Table  B-16.  Identification  of  Expenditures 


Area  of  Cost  Saving  or  Avoidance 

Determination  of  MOE 

Build 

The  simulation  is  a  product,  just  like 
a  weapon  system 

Treat  like  an  acquisition  program, 
important  to  do  a  feasibility  study 

Costs  of  “productizing”  M«&S; 
making  it  usable  by  several  people 

May  be  applicable  if  using  a  legacy 
simulation 

Integration  of  simulation  with  other 
simulations  or  hardware 

May  have  to  pay  to  modify  interface 
software  or  equipment 

Prove 

W&A 

Probably  50%  of  acquisition  costs, 
simulations  are  software  intensive 

Operator  acceptance — do  the  users 
and  customers  believe  the  results? 

Need  operator  involvement  from 
start,  increasing  acquisition  costs 

Validation  testing 

Actual  hardware  tests  to  validate  the 
models  may  include  live  ordnance 
firings — ^number  and  type  need  to  be 
determined  during  program  planning 

Use 

Training  of  users 

Recurring  cost — users  will  change 

Training  of  facilitators — people  who 
train  the  users  and  run  the 
simulation 

Recurring  cost  but  at  a  slower  pace 
than  training  of  users 

Computer  time 

Lease  or  ownership  costs 

Equipment  storage,  access  to  space 

May  be  leased 

Scheduling  time 

Delay  to  program  because 
simulation  facility  was  not  available 
exactly  when  needed 

Setup  costs 

Simulation  may  be  scenario 
dependent  or  user  tailored 

Duplication  of  equipment 

Users  may  choose  to  purchase  their 
own  systems — they  also  need  to 
duplicate  facilitator  costs  and 
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Feed 

Population  of  databases 

Dependent  upon  scenario,  access  to, 
and  cost  of  data 

Equipment  maintenance  contracts 

Recurring  costs 

Configuration  management 

Recurring  costs 

Depreciation  of  equipment 

Sometimes  applicable 

Lease  of  communications  lines 

Recurring  costs 

Update  of  databases 

Scenario  dependent 

Software  support  activity 

Make  modifications,  upgrade,  fix 

Revalidation 

Each  time  a  change  is  made 

Maintenance  of  libraries 

Baselines  and  distributing  releases 

Point  of  contact  for  questions 

Necessary  if  system  at  multiple  sites 
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ANNOTATED  BIBLIOGRAPHY 


This  bibliography  contains  not  only  the  documentation  used  in  writing  the  report 
but  other  material  collected  by  the  Task  Force  as  well.  In  some  cases,  material  is  taken 
verbatim  from  the  cited  government  document.  The  collection  and  abstraction  efforts  are 
continuing  under  the  long-term  Impact  Assessment  task  performed  by  DMSO. 


[Acquisition  Task  Force  on  M&S  1994] 

Acquisition  Task  Force  on  Modeling  and  Simulation,  Cost  and  Operational  Effectiveness 
Analysis  Focus  Group.  Report  on  Proceedings:  Utility  of  Synthetic  Environments  in 
Support  of  Operational  Test  and  Evaluation  1993. 

This  report  outlines  the  final  results  of  the  focus  group  convened  by  the  Acquisition  Task 
Force  on  Modeling  and  Simulation  to  address  the  uses  of  M&S  in  Cost  and  Operational 
Effectiveness  Analysis  (COEA).  The  participants  were  asked  to  look  beyond  the  confines 
of  the  COEA  process  for  ways  that  COEA  products  and  by-products  could  be  used  in 
non-traditional  products. 


[AEDC  1995] 

Arnold  Engineering  Development  Center.  DoD  Integrated  Test  and  Evaluation  Successes 
Using  Computing  Resources.  USAF  Arnold  Engineering  Development  Center,  October 
1995. 

This  paper  outlines  the  role  of  the  Arnold  Engineering  Development  Center  (AEDC)  in 
providing  knowledge  for  risk  management  and  decision  making  during  the  development 
and  operation  of  the  Center’s  modeling  system.  Using  an  Integrated  Test  and  Evaluation 
(IT&E)  approach  to  systems  support  development,  AEDC  combines  analysis  tools  with 
concurrent  ground  tests  of  multiple  subsystems  to  accelerate  and  improve  the  integration 
of  the  subsystems  before  flight.  Twenty-one  successes  using  IT&E  in  the  general  fields  of 
aerodynamics,  turbine  engines,  customer  support,  and  future  opportunities  are  presented 
with  estimated  cost  savings  for  selected  systems. 
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developed  as  the  “Post  Oak”  simulator  in  1979  and  renamed  to  TACSIM  in  1980.  Its 
initial  function  exercised  intelligence  missions  utilizing  scripted  scenario  databases  against 
enemy  forces  and  generating  reports  in  U.S.  Message  Text  Format  (USMTF).  According 
to  the  article,  TACSIM  development  continues  and  currently  supports  the  Distributed 
Interactive  Simulation  (DIS)  and  Aggregate-Level  Simulation  Protocol  (ALSP).  This  is  a 
thorough  article  covering  TACSIM  from  its  origin  in  1979  to  the  present.  It  contains 
detail  on  TACSIM’s  employment  and  lists  collection  assets,  but  remains  understandable  to 
those  outside  the  intelligence  community.  When  discussing  scripting,  sensor  flexibility, 
and  TACSIM  analysis,  the  article  focuses  on  the  function  and  responsibilities  of  military 
intelligence.  This  contributor  (TACSIM)  to  ALSP  is  an  integral  part  in  the  overall  success 
of  the  Confederation  of  Models  currently  incorporated  in  ALSP.  Here  is  a  pertinent 
article  that  portrays  this  simulation  in  an  easily  understood  manner. 
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History.” //w/waw  Factors  33  (1991);  343-362. 

SIMNET,  an  acronym  for  simulator  networking,  was  initiated  in  1983  as  a  project  on 
large-scale  simulator  networking  by  the  Defense  Advanced  Research  Projects  Agency. 
This  paper  summarized  the  technical  history  of  SIMNET  development  and  identifies 
lessons  learned  that  could  contribute  to  the  success  of  future  efforts  to  develop  training 
technologies  and  systems,  especially  for  collective  training.  It  concludes  with  a  discussion 
of  the  implications  and  challenges  of  SIMNET  for  the  human  factors  and  training 
technology  communities. 
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Training;  Military  Examples.”  In  Teams:  Their  Training  and  Performance,  R.  W.  Swezey 
and  E.  Salas,  eds.  Ablex,  Norwood,  NJ,  1992. 

The  authors  provide  an  overview  of  the  need  for  team  training  in  the  military  and  the  use 
of  simulators  to  deliver  this  training.  Examples  of  training  technologies  employing 
simulators  for  team  training  are  provided,  citing  Army,  Navy,  and  Air  Force  applications. 
Future  trends  are  noted. 


[Angieretal.  1993] 

B.  N.  Angier,  E.  A.  Alluisi,  and  S.  A.  Horowitz.  Simulators  and  Enhanced  Training. 
IDA  Paper  P-2672.  Institute  for  Defense  Analyses,  Alexandria,  VA,  1992. 

This  study  focuses  on  the  issue  of  whether  increased  use  of  individual  and  networked 
simulators  and  training  devices  holds  the  potential  to  maintain  military  manpower 
capability  during  a  period  of  declining  budgets  and  force  levels.  It  reviews  findings  of 
several  previous  studies,  highlighting  key  findings.  The  operating  costs  of  flight 
simulators  are  about  10%  of  actual  equipment  per  hour  trained;  if  acquisition  cost  is  taken 
into  account,  the  yearly  operating  cost  is  about  33%  of  actual  equipment.  The  majority  of 
tasks  trained  on  simulators  (59%)  have  significant  positive  transfer  to  flight  performance 
(transfer  effectiveness  ratios  greater  than  0.33).  The  authors  suggest  a  cautious  approach 
to  increased  use  of  simulation  in  lieu  of  flying  hours,  but  felt  that  it  would  not  be  overly 
risky  to  transfer  perhaps  5%  of  the  flying  hour  budget  to  simulator  acquisition  and 
operation.  Analysis  of  an  effectiveness  comparison  between  simulator  networking 
(SIMNET)  and  home-station  field  training  indicates  that  SIMNET  is  extremely  effective  in 
increasing  performance  for  SIMNET-trainable  tasks  relative  to  field  training.  Tradeoff 
analyses  shows  that  investment  in  SIMNET-like  facilities  could  be  repaid  by  an  8  to  14% 
decrease  in  operating  tempo  (OPTEMPO). 


[Armed  Forces  Journal  1993] 

“Training  &  Simulation:  Battleground  for  Digitized  Warriors.”  Armed  Forces  Journal 
International  132  (4)  (November  1993);  40-41. 

This  article  focuses  on  the  missions  of  the  Army’s  Simulation,  Training,  and 
Instrumentation  Command  (STRICOM),  headquartered  in  Orlando,  FL.  STRICOM  is 
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responsible  for  managing  DOD  technical  efforts  in  Distributed  Interactive  Simulation, 
networking  simulation  components  throughout  the  world,  and  developing  and  purchasing 
specialized  training  and  simulation  devices.  It  describes  STRICOM’s  efforts  in  linking 
battle  labs  and  battlefields,  and  concludes  with  a  brief  discussion  of  its  progress  in 
modeling  equipment  for  allies.  This  short  article,  which  provides  the  reader  with  a  basic 
knowledge  of  STRICOM,  educates  those  previously  unaware  of  this  command  and 
highlights  its  importance  to  DoD’s  efforts  to  link  training  and  testing  resources  throughout 
the  services. 


[ARI  1995a] 

Army  Research  Institute  for  the  Behavioral  and  Social  Sciences.  “ARI  Support  of 
Tactical  Engagement  Simulation.”  ARI  Newsletter  5  (1)  (1995):  1-15. 

This  special  issue  describes  ARI’s  key  contributions  to  the  development  of  tactical 
engagement  simulation  (TES).  It  begins  with  an  historical  overview  of  TES  and  MILES 
and  their  impact  on  the  evolution  of  Combat  Training  Centers  (CTCs).  Other  articles 
address  lessons  learned  and  the  value  of  virtual  and  constructive  simulation,  including 
Distributed  Interactive  Simulation  (DIS),  for  multi-service  and  Reserve  Component  (RC) 
tasks.  One  article,  for  example,  presents  data  supporting  the  value  of  DIS  for  multi- 
Service  Close  Air  Support  training.  Another  article  presents  data  showing  the  value  of 
JANUS  and  SIMNET  for  training  RC  units. 


[ARI  1995b] 

Army  Research  Institute  for  the  Behavioral  and  Social  Sciences.  “Simulator  Training 
Research  Advanced  Testbed  for  Aviation  (STRATA).”  ARI  Newsletter  5  (2)  (1995);  7- 
12. 

The  ARI  STRATA  facility  at  Ft.  Rucker,  AL,  provides  a  powerful  tool  for  collecting 
helicopter  pilot  training  and  cost-eflFectiveness  data.  This  article  summarizes  experiments 
conducted  since  the  arrival  of  STRATA  at  Ft.  Rucker  in  1992.  One  series  of  experiments 
validated  STRATA  for  simulating  AH-64H  helicopter  tasks.  A  second  series  tested 
alternative  training  in  a  traditional  lockstep  manner.  Experimentally  trained  crews  had 
higher  gunnery  scores,  sustained  less  battle  damage,  and  engaged  more  targets  than  the 
control  crews. 


[Army  Science  Board  1989] 

Army  Science  Board.  Close  Combat  (Heavy)  Training  Strategy  for  the  1990s.  U.S. 
Department  of  Defense,  Washington,  DC,  March  1989. 

The  Army  Science  Board  conducted  a  study  of  the  training  strategy  for  heavy  forces,  with 
a  focus  on  the  role  that  simulation  might  have  in  future  Army  training.  The  board 
expressed  the  opinion  that  it  would  be  possible  to  reduce  OPTEMPO  (operating  tempo) 
and  training  ammunition  by  15  to  20%  while  maintaining  the  same  or  a  better  level  of  unit 
performance,  provided  that  (1)  compensating  funds  would  be  provided  to  enhance  and 
operate  simulators,  (2)  simulators  could  be  tailored  to  the  special  needs  of  units,  and  (3) 
simulators  would  be  used  effectively.  The  reduction  assumed  a  base-level  tank  main  gun 
annual  ammunition  allocation  of  100  rounds  (active)  and  48  rounds  (National  Guard),  800 
miles  per  vehicle,  and  15.8  flight  hours  per  pilot  per  month. 


[Automotive  Industiy  Action  Group  1994] 

Automotive  Industry  Action  Group  (AIAG).  Solid  Modeling  White  Paper.  AIAG, 
Southfield,  MI,  June  1994. 

This  white  paper  is  the  result  of  a  request  from  the  AIAG  Original  Equipment 
Manufacturer  Computer-Assisted  Design/Computer-Assisted  Manufacturing  (OEM 
CAD/CAM)  Task  Force  that  a  work  group  composed  of  members  from  Chrysler,  Ford, 
and  General  Motors  be  formed  to  write  a  white  paper  on  solid  modeling.  Representatives 
from  each  of  the  three  automobile  manufacturers  provided  information  concerning  their 
past  and  present  experience  with  solid  modeling,  as  well  as  indications  of  their  strategies 
for  the  future. 


[Automotive  Engineering  1994a] 

“Driving  Simulation  at  Ford.”  Automotive  Engineering  (September  1994):  37-40. 

This  article  reviews  Ford’s  experience  in  using  simulation  in  studying  driver  performance 
in  the  same  controlled  manner  as  simulation  is  used  in  studying  mechanical  vehicle 
components.  The  information  was  supplied  by  Jeflfiy  A.  Greenberg  and  Thomas  J.  Park  of 
the  Ford  Motor  Co.  Research  Laboratory. 


[Automotive  Engineering  1 994b] 

“Driving.”  Automotive  Engineering  \99A):  14-19. 

The  role  of  driving  simulation  in  the  vehicle  design  process  at  General  Motors  (GM)  is 
reviewed  in  this  article.  The  article  suggests  that  the  GM  driving  simulator  is  a  viable  tool 
for  studies  performed  in  the  on-center  region  of  vehicle  performance.  Additional 
conclusions  and  recommendations  for  using  driving  simulators  are  presented.  Information 
for  this  article  was  supplied  by  Gary  P.  Bertollini,  Charles  M.  Johnson,  James  W.  Kuiper, 
James  C.  Kukula,  Malgorzata  A.  Robzveka,  and  William  E.  Thomas,  all  of  GM. 


[Bailey  1995] 

M.  Bailey.  Value  of  Modeling  and  Simulation.  Memorandum  to  the  M&S  Benefits  Task 
Force.  Naval  Air  Warfare  Center,  Weapons  Division  (NAWCWD),  China  Lake,  CA,  May 
1995. 

Bailey  writes  that  the  “beef’  of  M&S  “resides  in  the  intelligent  and  informed  application  of 
modeling  and  simulation  to  reduce  risk,  avoid  safety/environmental/security  issues,  and 
increase  the  quality  of  training  and  equipment.”  She  concludes  that  M&S  is  a  tool  of 
substantial  assistance  to  the  military  when  properly  applied. 


[Bailey  and  Kodak  1994] 

S.  S.  Bailey  and  G.  W.  Kodak.  Live  Fire  Versus  Simulation:  A  Review  of  the  Literature. 
NAWCTSD-SR-94-002.  Naval  Air  Warfare  Center,  Training  Systems  Division,  Orlando, 
FL,  1994. 

The  authors  reviewed  several  studies  relating  to  the  use  of  simulation  in  lieu  of  live  fire  for 
the  purpose  of  reducing  the  cost  of  training  Marine  Corps  personnel.  Ammunition  costs 
for  training  Marines  range  from  $1,762  (MOS  0311,  rifleman)  to  $19,145  (MOS  0352, 
TOW  II  gunner).  Many  of  the  empirical  studies  have  demonstrated  that  performance  with 
simulation  is  at  least  equal  to  live  fire  training,  but  that  cost  is  lower.  The  Multipurpose 
Arcade  Combat  Simulator  (MACS)  is  used  to  augment  training  on  the  Ml 6  rifle  and  small 
weapons.  Effectiveness  studies  did  not  find  a  statistically  significant  difference  between 
live  fire  and  simulation,  but  soldiers  with  MACS  training  expended  less  rounds  during  live- 
fire  qualifications  and  fewer  soldiers  failed  to  qualify  when  compared  to  those  trained 
using  traditional  methods.  The  Weaponeer  is  a  part  task  trainer  for  training  on  a  variety  of 
weapons,  including  Ml 6,  SAW,  M60,  AT-4,  and  M203  grenade  launcher.  Weaponeer 


can  be  used  to  predict  and  record  fire  performance,  but  no  data  are  available  on  its  training 
effectiveness.  The  Squad  Engagement  Training  System  (SETS)  is  used  to  provide 
marksmanship  and  tactical  training  for  up  to  and  including  squad  level.  Several  studies 
have  shown  positive  transfer  from  SETS  to  live  fire.  The  Indoor  Simulated  Marksmanship 
Trainer  (ISMT)  is  used  to  train  on  several  small  arms  and  has  been  demonstrated  to 
benefit  live-fire  performance.  The  Precision  Gunnery  Training  System  (PGTS)  is  an 
inexpensive  trainer  for  the  TOW  and  Dragon  missiles,  whose  rounds  are  prohibitively 
expensive  ($11,500  and  $19,145,  respectively,  per  round)  to  fire  in  training  exercises. 
PGTS  has  been  demonstrated  to  be  cost  effective,  and  also  permits  training  that  would 
otherwise  cost  several  hundred  million  dollars  per  year  if  actual  missiles  were  used. 


[Baker  et  al.  1994] 

E.  R.  Baker,  L.  Cooper,  B.  A.  Carson,  and  A.  E.  Stevens.  “Software  Acquisition 
Management  Maturity  Model  (SAM3).”  Program  Manager  23  (4)  (July-August  1994); 
43-48. 

This  article  describes  SAM3  and  its  underlying  concepts.  SAM3  is  a  hierarchical  structure 
of  Key  Process  Areas,  Key  Practices,  and  Key  Indicators.  It  is  organized  into  five  levels 
of  maturity:  Initial,  Repeatable,  Defined,  Managed,  and  Optimizing.  The  acquisition 
management  maturity  model  is  the  basis  for  assessments  of  the  acquisition  management 
capabilities  of  the  organization.  This  article  provides  a  clear  explanation  for  a  complex 
model.  Graphics  add  to  the  presentation  and  greatly  assist  the  reader  in  understanding  the 
model. 


[Bell  and  Crane  1992] 

H.  H.  Bell  and  P.  M.  Crane.  Training  Utility  of  Multiship  Air  Combat  Simulation.  USAF 
Armstrong  Laboratory,  Mesa,  AZ,  1992. 

The  authors  describe  evaluations  with  developmental  distributed  interactive  simulation 
(DIS)  systems  designed  to  support  multiship  air  combat  training  in  a  combat  engagement 
simulation  environment.  In  the  first  evaluation,  the  simulator  realistically  represents  the 
threat  environment  (e  g.,  two  F-15  cockpits,  visual  world,  enemy  surface-to-air  and 
electronic  jamming,  enemy  aircraft,  pilots  and  air  weapons  controllers).  Responses  of 
participating  pilots  and  air  weapons  controllers  indicate  that  they  believed  that  some 
mission  areas  were  better  trained  in  the  simulator  than  in  unit  training;  multibogey, 
reaction  to  SAM,  dissimilar  air  combat  tactics,  employment  of  electronic  countermeasures 
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(ECM),  all-aspect  defense,  escort  tactics,  all-weather  employment,  communications 
jamming,  low-altitude  tactics,  threat  warning  assessment,  and  work  with  air  weapons 
controller.  Mission  areas  where  simulator  training  was  rated  inferior  to  unit  training  were 
visual  lookout,  tactical  formation,  visual  identification,  and  mutual  support. 


[Berg  et  al.  1993a] 

R.  M.  Berg,  A.  M.  Adedeji,  and  G.  W.  Steadman.  Simulation  Offset  to  Live  Fire 
Training  Phase  2  Results:  Application  of  the  At  Least  Equal  Effectiveness  Methodology 
to  Simulator  Use  in  Marine  Corps  Infantry  Training  Programs.  CRM-93-112.  Center 
for  Naval  Analyses,  Alexandria,  VA,  1993. 

This  study  addresses  this  question:  “To  what  extent  does  it  make  sense— from  both 
training  effectiveness  and  cost  perspectives — ^for  the  Marine  Corps  to  use  simulators  in 
performing  infantry  training  tasks  that  are  now  done  predominantly  with  live-fire?”  The 
study  concludes  that  third-generation  simulators  such  as  Indoor  Simulated  Marksmanship 
Trainer  (ISMT)  can  be  used  cost  effectively  in  USMC  live-fire  training  programs,  that 
procuring  them  would  be  a  very  good  investment,  that  they  would  increase  the  overall 
quality  and  effectiveness  of  training,  and  significantly  reduce  the  total  annual  cost  of 
training.  The  authors  recommend  that  the  USMC  “proceed  expeditiously  to  implement 
this  type  of  system  into  its  infantry  training  programs.” 


[Berg  et  al.  1993b] 

R.  M.  Berg,  A.  M.  Adedeji,  and  C.  Trenholm.  Simulation  Offset  to  Live  Fire  Training 
Study:  Assessment  of  Marine  Corps  Live  Fire  Training  Support.  CIM-238.  Center  for 
Naval  Analyses,  Alexandria,  VA,  1993. 

The  USMC  has  traditionally  emphasized  live-fire  training  and  placed  low  priority  on 
acquiring  simulators.  Anticipated  reductions  in  training  resources  led  to  this  study  of  the 
potential  use  of  simulators  in  lieu  of  live  training.  The  study  examines  simulators  currently 
used  in  the  USMC  and  Army  and  the  potential  for  expanded  use  in  the  USMC.  The 
training  of  infantry  marksmanship  is  estimated  to  be  the  single  most  expensive  training 
program  in  the  USMC.  M16A2  rifle  qualification  requires  each  Marine  to  fire  250  rounds 
on  the  rifle  range.  The  biggest  cost  driver  in  marksmanship  training  is  the  cost  of  training 
ammunition.  Additional  significant  costs  are  involved  in  operating  and  maintaining  ranges. 
Simulators,  such  as  the  Multipurpose  Arcade  Combat  Simulator  (MACS),  have  the 
potential  to  significantly  reduce  these  costs  in  the  USMC. 
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[Bessemer  1991] 

D.  W.  Bessemer.  Transfer  of  SIMNET  Training  in  the  Armor  Officer  Basic  Course.  ARI 
TR  920.  USA  Research  Institute  Field  Unit— Ft.  Knox,  Ft.  Knox,  KY,  1991 . 

Bessemer  describes  a  quasi-experimental  assessment  of  transfer  of  tactical  training  from 
SIMNET  to  field  training.  Baseline  classes  without  simulator  training  were  compared  in 
an  interrupted  time  series  design  to  classes  with  simulator  training.  Transfer  was  found 
using  indicators  of  (1)  the  amount  and  type  of  field  training,  (2)  platoon-level  officer 
leadership  performance,  and  (3)  overall  tactical  leadership  qualities  just  prior  to 
graduation.  Benefits  of  simulator  training  increased  in  successive  classes  as  instructors 
learned  to  train  with  simulators  more  effectively. 


[Boldovici  and  Bessemer  1994] 

J.  A.  Boldovici  and  D.  W.  Bessemer.  Training  Research  with  SIMNET:  Lessons  Learned 
from  Simulation  Networking.  ARI  Technical  Report  1006.  USA  Research  Institute  Field 
Unit— Ft.  Knox,  Ft.  Knox,  KY,  1994. 

The  authors  provide  a  critique  of  methods  commonly  used  in  field  experiments  to  evaluate 
training  systems.  In  a  review  of  several  reports  of  training  effectiveness  research  as 
applicable  to  evaluation  of  the  Close  Combat  Tactical  Trainer  (CCTT),  the  authors 
conclude  that  one-shot  empirical  evaluations  of  the  kind  needed  to  meet  acquisition,  test, 
and  evaluation  regulations  (e.g.,  AR  70-1,  TRADOC  regulations  71-9  and  350-4)  are 
costly  and  often  so  flawed  by  compromises  in  research  design  that  they  are  unlikely  to 
meet  evaluation  objectives.  Common  problems  are  insufficient  statistical  power, 
inadequate  sampling,  inappropriate  analyses,  inadequate  controls,  and  lack  of  control  data. 
The  authors  recommend  that  evaluators  consider  alternatives  to  empirical  evaluations  such 
as  in-device  learning  experiments,  quasi-experimental  designs,  and  correlational  methods. 


[Boldovici  et  al.  1985] 

J.  A.  Boldovici,  D.  W.  Bessemer,  and  D.  F.  Haggard.  Review  of  the  Ml  Unit-Conduct  of 
Fire  Trainer  (UCOFT^  Validation  and  Verification  Test  Report.  ARI  Research  Note  85- 
56.  USA  Research  Institute  Field  Unit — ^Ft.  Knox,  Ft.  Knox,  KY,  1985. 

This  review  was  performed  at  the  request  of  the  then-Under  Secretary  of  the  Army,  James 
Ambrose.  It  examines  the  somewhat  mixed  evidence  contained  in  the  UCOFT  Validation 
and  Verification  (VA^)  test  report,  and  concludes  that  the  UCOFT  provided  improvements 
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in  gunner  proficiency  on  the  UCOFT  under  a  number  of  different  conditions,  although  the 
WfV  did  not  demonstrate  transfer  to  operational  equipment.  Substantial  gains  were  found 
in  percents  of  targets  acquired,  engaged,  hit,  and  killed  for  groups  undergoing  sustainment 
and  transition  training.  Gains  were  attributed  to  improvements  in  acquisition  time, 
engagement  time,  and  first-round  hits,  which  in  turn  allowed  time  to  scan,  acquire,  and 
engage  available  second  and  third  targets. 


[Brown  1992] 

F.  J.  Brown.  Battle  Command  Staff  Training.  Institute  for  Defense  Analyses,  Alexandria, 
VA,  December  1992. 

The  objective  of  this  paper  is  to  design  and  develop  a  new  distributed  simulation-based 
intensified  training  readiness  strategy  for  the  Reserve  Component.  The  author  introduces 
“...an  innovative  prototype  for  training  battle  staff  synchronization  using  advanced 
distributed  simulation.”  This  innovative  prototype  is  called  Battle  Command  Staff 
Training  (BCST).  Its  focus,  on  the  battalion  and  brigade  levels,  shows  how  this  new 
training  strategy  will  positively  affect  the  Reserve  Component  unit.  In  their  day  (late 
1970s,  early  1980s),  these  simulation  training  devices  were  novel.  Today,  the  state  of  the 
art  is  generations  beyond  the  battalion  and  brigade  simulations  presented  here.  This  brief 
paper,  with  appendices  and  supplementary  tables,  addresses  the  value  of  distance  learning. 
It  asserts  that  distance  learning  is  the  wave  of  the  future.  The  author  proposes  BCST  as  a 
new  model  for  training.  His  use  of  the  term  “model”  refers  to  structure,  rather  than  a 
computer  program.  The  simulation-based  strategy  to  which  he  refers  is  simply  the 
leveraging  of  existing  battalion  and  brigade  simulations  that  are  used  extensively  by  the 
Active  Component. 


[Brown  et  al.  1988] 

R.  E.  Brown,  R.  G.  Pishel,  and  L.  D.  Southard.  Simulation  Networking.  TRAC-WSMR- 
TEA-8-99.  USA  TRADOC  Analysis  Command,  White  Sands  Missile  Range,  NM,  1988. 

SIMNET  effectiveness  as  a  platoon-level  tactical  training  device  is  evaluated  by 
comparing  before  vs.  after  training  performance  results  for  a  SIMNET-trained  group  and  a 
field-trained  control  group.  Both  training  groups  consist  of  four  Ml  tank  platoons. 
Principal  results  are  that  SIMNET  training  increased  field  exercise  platoon  performance, 
command  and  control,  and  leadership,  and  adequately  portrayed  vehicle  and  battlefield 
sounds.  SIMNET  is  reported  to  be  ineffective  for  training  tasks  related  to  obstacles. 


dismounting  troops,  air  attack,  and  use  of  smoke.  The  visual  display  is  inadequate  for 
viewing  objects  beyond  1,000  meters  and  for  identifying  hull/turret-down  defiled 
positions.  The  simulated  Ml  tank  speed  is  unrealistic.  SIMNET  also  improved 
performance  of  command  and  control,  platoon  movement,  leadership,  and  fire  distribution 
during  the  company  team  Army  Training  and  Evaluation  Program  (ARTEP). 


[Bryant  et  al.  1992] 

J.  A.  Bryant,  N.  L.  Lewis,  M.  Stapp,  A.  A.  Zamarripa,  J.  Cox,  J.  Wilhelm,  and  M.  Walker. 
JANUS(A)  Brigade/Battalion  Simulation  Cost  and  Training  Effectiveness  Analysis. 
TRAC-WSMR-CTEA-92-006.  USA  TRADOC  Analysis  Command,  White  Sands  Missile 
Range,  NM,  1992. 

This  Cost  and  Training  Effectiveness  Analysis  assesses  the  command  and  control  portion 
of  the  Simulator/Simulation-Based  Training  Program  Analysis  (SIM2)  and  the 
Battalion/Brigade  Battle  Simulation  (BBS),  based  on  analyses  and  opinion  data  gathered 
fi'om  subject  matter  experts  and  supported  by  a  literature  review  to  provide  historical  data 
on  the  simulations.  The  literature  review  indicates  that  JANUS(A)  is  effective  in  training 
company-level  officers  and  platoon  leaders  on  current  tactics  and  doctrine  and  that  BBS  is 
effective  at  the  brigade/battalion  level.  Study  results  indicate  that  a  large  percentage  of 
mission  training  plan  tasks  can  be  trained  in  the  Close  Combat  Tactical  Trainer  (CCTT), 
JANUS(A),  BBS,  and  CCTT  have  somewhat  overlapping,  but  also  individually  unique 
training  capabilities.  Based  on  a  15-year  life  expectancy,  the  hourly  cost  of  using 
JANUS(A)  was  estimated  to  be  $163. 


[Burnside  1990] 

B.  L.  Burnside.  Assessing  the  Capabilities  of  Training  Simulations:  A  Method  and 
Simulation  Network  (SIMNET)  Application.  ARI  Research  Report  1565.  USA  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  Alexandria,  VA,  1990. 

Burnside  describes  a  method  developed  for  assessing  Army  Training  and  Evaluation 
Program  (ARTEP)  Mission  Training  Plan  (MTP)  standards  that  can  be  met  and  subtasks 
and  tasks  that  can  be  performed  when  conducting  trmning  with  TADSS.  The  method  is 
applied  to  assess  the  performance  capabilities  of  SIMNET.  Using  the  criterion  that  tasks 
must  be  at  least  partially  performable  to  be  trainable  in  SIMNET,  the  study  finds  that  35% 
of  ARTEP  MTP  tasks  can  be  trained  with  SIMNET.  The  platoon  echelon  has  the  highest 
percentage  of  trainable  tasks  (41%)  and  of  tasks  not  supported  by  the  simulation  (46%). 
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The  author  concludes  that  the  method  provides  an  efficient  means  to  assess  the  capabilities 
and  requirements  of  TADS  S. 


[Case  1991] 

F.  T.  Case.  Analysis  of  Air  Operations  During  Desert  Shield/Desert  Storm.  AFSAA 
Report  No.  28467;  DTIC  AD-B161  849.  Air  Force  Studies  and  Analysis  Agency, 
Department  of  the  Air  Force,  Washington,  DC,  November  1991 . 

No  abstract  available. 


[Center  for  Naval  Analyses  1995] 

Center  for  Naval  Analyses.  Developing  a  Navy  Strategy  for  Modeling  and  Simulation: 
Final  Report.  Center  for  Naval  Analyses,  Alexandria,  VA,  March  1995. 

This  report  develops  a  Navy  strategy  for  modeling  and  simulation  through  a  series  of  20 
recommendations  regarding  Navy  modeling  and  simulation.  These  recommendations, 
which  are  first  presented  in  numerical  order  with  a  brief  explanation,  are  then  grouped 
under  the  areas  of  the  Navy  modeling  community,  distributed  simulation,  acquisition 
support,  test  and  evaluation,  training,  and  verification,  validation  and  accreditation.  The 
second  grouping  includes  conclusions,  recommendations,  and  justifications  for  each  of  the 
20  points.  The  report  presents  some  valid  recommendations  in  a  number  of  key  areas  in 
the  modeling  and  simulation  community.  Although  the  report  purports  a  plan  of  action  to 
achieve  each  of  the  recommendations  expressed,  it  frequently  provides  insufficient  detail 
to  validate  the  recommendation.  The  summary  of  key  study  conclusions  discusses  the 
need  for  centralized  coordination  of  modeling  development,  and  the  conclusion  that 
modeling  and  simulation  can  contribute  meaningfully  to  the  quality  of  Navy  acquisition, 
test  and  evaluation,  and  training.  The  most  meaningful  topic  for  training  and  education  is 
contained  in  Appendix  E,  Training  Applications. 


[Crane  and  Berger  1993] 

P.  M.  Crane  and  S.  C.  Berger.  Multiplayer  Simulator  Based  Training  for  Air  Combat. 
USAF  Armstrong  Laboratory,  Williams  AFB,  AZ,  1993. 

The  authors  describe  Training  Requirements  Utility  Evaluation  (TRUE),  a  series  of 
simulated  air  combat  exercises  conducted  in  Multiship  Research  and  Development 
Program  (MULTIRAD),  a  SIMNET-compatible  an  air  combat  simulator.  TRUE  was 
conducted  to  identify  mission  tasks  and  skills  that  can  be  effectively  trained  using 


multiplayer  simulation.  Pilots  received  training  on  MULTIRAD  and  then  rated  their 
interest  in  receiving  additional  training  on  each  of  30  tasks.  The  five  tasks  with  the  lowest 
rated  interest  are  primarily  visual  tasks.  Tasks  with  the  highest  rated  interest  can  usually 
be  practiced  only  in  large  exercises  or  cannot  be  practiced  except  in  simulation.  It  was 
concluded  that  multiplayer  simulator-based  training  is  a  valuable  training  medium  for 
increasing  wartime  readiness,  especially  for  less  experienced  pilots. 


[Davis  1991] 

P.  Davis.  Letter  included  in  the  Congressional  Record  Volume,  “The  CFE  Treaty,”  from 
hearing  before  the  Subcommittee  on  European  Affairs  Committee  of  Foreign  Relations, 
U.S.  Senate,  March  20,  1991;  July  11,  16,  17,  and  25,  1991,  pp.  327  and  ff. 

In  a  response  to  a  Task  Force  request  for  information,  Davis  recommends  the  above 
citation  as  summarizing  his  perception  of  RAND  work  with  simulation  and  the  RAND 
Strategy  Assessment  System  (RSAS)  in  affecting  the  CFE  (Conventional  Forces  in 
Europe)  negotiations.  Additionally,  he  points  out  that  the  following  could  be  claimed; 
“(1)  Analysis  based  on  simulation-based  analysis  conflicting  with  conventional  wisdom 
strongly  influenced  NATO’s  insistence  on  highly  as5mimetric  reductions  toward  equal 
ceilings;  (2)  analysis  for  the  Office  of  the  Secretary  of  Defense  (OSD)  clarified  the 
circumstances  under  which  deep  reductions  would  and  would  not  be  militarily 
destabilizing,  even  with  nominal  ceilings;  and  (3)  analysis  for  OSD,  supported  by 
simulation-based  work,  helped  clarify  and  orient  the  US  sense  of  objectives  for  confidence 
and  security-building  measures.”  Davis  also  recommends  a  paper  by  Jim  Thompson  and 
Nanette  Gantz  that  had  a  major  effect  in  determining  NATO’s  position  in  the  CFE 
negotiations.  Davis  points  out  that  Thompson  and  Gantz’s  work,  circa  1988,  relied  on 
simulation  results  from  RSAS. 


[Defense  Science  Board  1988] 

Defense  Science  Board.  Report  of  the  Defense  Science  Board  Task  Force  on  Computer 
Applications  to  Training  and  Wargaming.  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Washington,  DC,  1988. 

TMs  Defense  Science  Board  (DSB),  an  advisory  committee  of  senior  military  analysts 
chdred  by  Dr.  Anita  Jones,  focused  on  the  training  of  Joint  operational  commanders,  their 
staffs,  and  the  commanders  and  staffs  who  report  to  them.  The  report  stated  that 
computer-based,  simulated  scenarios  offer  the  only  practical  and  affordable  means  to 


improve  the  training  of  Joint  operational  commanders,  their  staffs,  and  the  commanders 
and  staffs  who  report  to  them.  Such  decision  makers  need  the  opportunity  to  exercise 
their  decision  skills,  to  test  war  plans,  and  to  train  to  work  as  a  closely  coordinated  force. 
Increasingly,  Joint  training  cannot  be  conducted  in  the  anticipated  theater  of  operations. 
There  are  political  objections  to  disruption  of  civil  activity.  The  cost  of  an  actual  exercise 
at  this  level  is  great.  Battle  simulation  offers  the  only  opportunity  to  practice  the  use  of 
certain  weapon  systems,  sensors,  tactics,  and  techniques  against  a  skilled  adversary.  The 
report  recommended  that  the  Chairman,  Joint  Chiefs  of  Staff  (CJCS)  make  Joint 
simulations  interoperable;  promote  their  usage;  and  establish  requirements  for  future 
capabilities,  a  prototype  program,  and  undertake  a  major  joint  training  initiative. 


[Defense  Science  Board  1993] 

Defense  Science  Board.  Report  of  the  Defense  Science  Board  Task  Force  on  Simulation, 
Readiness  and  Prototyping.  Office  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Washington,  DC,  1993. 

This  Defense  Science  Board,  an  advisory  committee  of  senior  military  analysts  co-chaired 
by  Dr.  Joseph  Braddock  and  General  Maxwell  Thurman,  USA  (Ret.),  focused  on  the 
effect  of  Advanced  Distributed  Simulation  (ADS)  technology  on  Service  and  Joint 
readiness.  The  task  force  declared  the  belief  that  ADS  technology  can  greatly  improve 
training  and  readiness,  will  help  expedite  prototyping,  and  can  transform  the  acquisition 
process.  The  belief  was  based  on  a  judgment  concerning  confidence.  The  task  force 
found  that  the  warfighting  community  has  embraced  ADS  and  is  extending  its  utility; 
warfighters  are  applying  distributed  and  multiple  simulations  methods  to  improve 
planning,  training,  and  mission  rehearsal.  They  have  developed  the  confidence  to  use  the 
technology  to  prepare  for  war.  In  contrast,  the  requirements  and  development  community 
is  using  less  powerful  simulation  techniques  and  the  acquisition  process  is  being 
handicapped. 


[Defense  Science  Board  1994] 

Defense  Science  Board.  Report  of  the  Defense  Science  Board  Task  Force  on  Readiness. 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Washington, 
DC,  1994. 

This  Defense  Science  Board,  an  advisory  committee  of  senior  military  analysts  chaired  by 
General  Edward  Meyer,  USA  (Ret.),  reviewed  the  state  of  readiness  of  the  Armed  Forces. 


The  report  made  the  following  remarks  about  the  relationship  of  M«&S  to  readiness. 
“Today,  M&S  offers  great  potential  as  an  affordable  and  effective  means  by  which  Joint 
forces  can  achieve  and  maintain  expertise  in  operational  and  tactical  tasks,  such  as 
employing  operational  firepower,  conducting  strategic  deplojmient,  employing  forces, 
developing  theater  intelligence,  conducting  mission  rehearsal,  and  operational  movement 
and  maneuver.  In  the  future,  technologies  of  the  ‘Information  Age’  offer  the  prospect  of 
making  M&S  increasingly  more  useful  in  enhancing  Joint  force  readiness.  Both  prudence 
and  economy  dictate  that  DoD  capitalize  on  Advanced  Distributed  Simulation  (ADS) 
technology  to  prepare  for  Joint  and  Combined  warfare  in  an  uncertain  world.  ADS  can 
provide  the  wherewithal  for  Joint  Task  Forces,  and  in  particular  Joint  tasks  for  staffs,  to 
practice  more  often  and  build  confidence.  Simulations  offer  the  potential  for  markedly 
improving  Joint  requirements  definition  and  refinement;  Joint  doctrine  development  and 
acquisition;  test  and  evaluation;  planning  and  course-of-action  assessments  or  rehearsals; 
and  military  education.  Live  exercises  and  training,  particularly  those  conducted  on 
instrumented  ranges  (e.g..  National  Training  Center,  29  Palms,  Fallon,  Nellis,  etc.),  will 
continue  to  provide  the  critical  component  of  unit  training.  In  recent  years,  however,  the 
Services  have  exploited  M&S  technology  to  enhance  individual  and  unit  readiness  while 
reducing  overall  training  costs.” 


[Deitchman  1988] 

S.  J.  Deitchman.  Preliminary  Exploration  of  the  Use  of  a  Warfare  Simulation  Model  to 
Examine  the  Military  Value  of  Training.  IDA  Paper  P-2094.  Institute  for  Defense 
Analyses,  Alexandria,  VA,  1988. 

This  paper  is  a  preliminary  examination  of  how  to  assess  the  military  value  of  unit  training 
in  the  same  quantitative  cost  and  effectiveness  terms  used  to  assess  investments  in  other 
areas  such  as  acquisition  of  new  weapon  systems  or  of  more  forces  of  various  kinds.  The 
military  value  of  training  in  relation  to  the  value  of  weapons  or  forces  is  difficult  to 
quantify,  and  data  that  would  shed  light  on  it  are  sparse;  a  new  approach  is  proposed  here. 
The  approach  consists,  first,  of  experimenting  with  a  large-scale  computer  simulation  of 
warfare  to  find  how  imputed  effects  of  unit  training  on  unit  proficiency  affect  the  outcome 
of  a  modeled  NATOAVarsaw  Pact  conventional  conflict.  Second,  the  outputs  of  the  model 
runs  are  reviewed  for  reasonableness,  and  are  used  to  set  up  a  series  of  questions  to  elicit 
military  judgments  and  any  available  test.  The  final  steps  will  be  to  estimate  the  costs  of 
the  levels  and  kinds  of  training  that  emerge,  and  then  to  use  similar  modeling  techniques 
(or  existing  results)  to  compare  investments  in  training  and  in  force  size  or  force 


modernization  to  achieve  equal  effects  on  the  outcome  of  the  war.  This  paper  describes 
the  first  experiments,  the  results  of  the  initial  model  runs,  and  the  next  steps  to  be  taken  in 
the  investigation,  in  terms  of  questions  to  be  answered  by  military  experts,  data  to  be 
gathered  if  they  are  available,  analyses  to  be  performed  subsequently,  and  kinds  of  training 
that  emerge.  Then,  similar  modeling  techniques  (or  existing  results)  are  used  to  compare 
investments  in  training  and  in  force  size  or  force  modernization  to  achieve  equal  effects  on 
the  outcome  of  the  war. 


[Deitchman  1990] 

S.  J.  Deitchman.  Further  Explorations  in  Estimating  the  Military  Value  of  Training. 
IDA  Paper  P-2317.  Institute  for  Defense  Analyses,  Alexandria,  VA,  1990. 

This  report  describes  the  continuation  of  work  started  earlier  to  find  ways  of  quantifying 
the  military  value  of  training.  The  earlier  work  used  a  large-scale  simulation  model  of 
warfare  to  examine  the  potential  effects  of  assumed  force  improvements  imputed  to 
training  on  the  outcome  of  a  postulated  war  in  Europe.  The  work  described  in  this  report 
gathered  some  of  the  available  data  showing  the  effects  of  training  on  force  effectiveness, 
and  it  estimated  the  cost  of  training,  in  the  areas  of  platoon-size  armored  combat  and 
bombing  accuracy  by  tactical  attack  aircraft.  These  results  were  compared  with  the  results 
of  analyses  describing  the  effects  of  equipment  improvement  in  the  same  areas  of  unit 
combat.  The  report  shows  the  size  of  the  effects  in  each  case,  and  it  evaluates  the  relative 
contributions  of  training  and  hardware  advances  to  improvement  of  force  effectiveness 
and  the  relative  costs  of  the  two  approaches.  Conclusions  are  reached  about  preferred 
approaches  to  such  evaluations,  and  desirable  future  elaborations  of  the  research  are 
outlined. 


[Deitchman  1993] 

S.  J.  Deitchman.  Quantifying  the  Military  Value  of  Training  for  System  and  Force 
Acquisition  Decisions:  An  Appreciation  of  the  State  of  the  Art.  IDA  Paper  P-2881. 
Institute  for  Defense  Analyses,  Alexandria,  VA,  1993. 

Deitchman  reviews  the  state  of  knowledge  regarding  the  contribution  of  collective  training 
in  military  units  to  success  in  battle.  The  review  of  the  “sparse  relevant  literature” 
indicates  that  unit  training  under  realistic  conditions  can  increase  key  military  capabilities 
of  units  from  platoon  to  battalion  size  (and  equivalents  in  the  air  forces)  by  factors  of  2  on 
average.  Hardware  advances  can  increase  military  capability  by  like  amounts  if  the 
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requisite  unit  training  is  also  provided.  Without  the  investment  in  unit  training,  the 
capability  of  a  military  unit  should  be  discounted  by  about  50%. 


[Deluca  1993] 

R.  Deluca.  “Simulator  Builders  Boost  Art  in  Courting  Diversion  Seekers.”  National 
128  (November  1993):  19-21. 

This  article  focuses  on  the  expansion  of  traditional  defense  simulator  technologies  into  the 
commercial  entertainment  field  and  the  reasons  behind  this  shift.  It  cites  the  finite  market 
of  defense  simulation  sales,  the  wide-open  commercial  market,  reduced  delivery  costs,  and 
rapid  improvements  in  hardware  and  software  that  make  this  challenging  shift  attractive. 
The  article  continues  by  discussing  futuristic  landscapes,  the  proliferation  of  applications, 
as  well  as  current  commercial  constraints  and  solutions  to  networking  problems.  The 
article  approaches  simulation  from  a  slightly  different  angle  than  one  would  expect  to  find 
in  National  Defense  magazine:  its  subject  is  commercial  simulation.  It  describes  the 
concerns  of  the  commercial  simulation  industry  and  the  shift  toward  involvement  in  the 
field  of  entertainment.  This  article  is  of  interest  to  those  involved  in  defense  simulations 
and  illustrates  a  direct  relationship  between  defense  and  entertainment  applications. 


[Department  of  the  Air  Force  1993] 

Department  of  the  Air  Force.  Response  to  Air  Force  Simulation  Questions.  Unpublished 
manuscript.  USAF,  Washington,  DC,  1993. 

No  abstract  available. 


[Department  of  the  Air  Force  1995] 

Department  of  the  Air  Force.  “Simulation  and  Cost  Savings.”  Issues  in  Air  Force 
Simulation  and  Analysis  (February  1995). 

The  author  proposes  that  simulation  should  be  used  early  in  the  acquisition  process  in 
order  to  “save  time,  decrease  life  cycle  cost,  and  decrease  risk.”  The  author  further  states 
that  “the  use  of  prototyping  and  simulation  for  the  integration  of  the  Multi-mission 
Advanced  Tactical  Terminal  (MATT)  saved  between  one  and  three  years  of  acquisition 
time.” 
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[Department  of  the  Army  1993] 

Department  of  the  Army.  Simulations  and  Their  Relationship  to  OPTEMPO/Training 
Ammunition.  Unpublished  memorandum.  USA,  Washington,  DC,  1993. 

Virtual  simulation  is  being  used  to  reduce  UPTEMPO  (operating  tempo),  typically  by 
reducing  flying  hours  and  the  amount  of  training  ammunition.  An  example  is  flight 
simulators  for  Cobras,  Chinooks,  Blackhawks,  and  Apaches.  Air  crew  training  manuals 
permit  two  flying  hours  per  month  to  be  replaced  by  simulators.  Cost  avoidance  for  the 
active  component  due  to  flight  simulators  is  $68M  annually.  Fielding  of  the  Conduct  of 
Fire  Trainer  (COFT)  reduced  the  annual  tank  main  gun  ammunition  allocation  from  134  to 
100  rounds.  This  resulted  in  an  annual  cost  avoidance  of  approximately  $29M.  It  was 
predicted  that  future  fielding  of  the  Tank  Weapons  Gunnery  Simulation  System  (TWGSS) 
FY96-001  would  further  reduce  annual  ammunition  allocations  from  100  to  78,  an 
additional  cost  avoidance  of  $21M  per  year. 


[Department  of  the  Army  Operational  Evaluation  Command  1993] 

Department  of  the  Army  Operational  Evaluation  Command.  Study  Report:  Utility  of 
Synthetic  Environments  in  Support  of  Operational  Test  and  Evaluation.  USA, 
Alexandria,  VA,  1993. 

This  study  reports  on  an  experiment  conducted  by  the  Army  to  evaluate  the  improved 
capability  of  the  M1A2  tank  versus  the  MlAl  tank  and  to  examine  the  utility  of  synthetic 
environments  for  support  of  Operational  Test  and  Evaluation  (OT&E).  The  results  of  the 
study  support  the  evaluation  of  the  utility  of  synthetic  environments  in  support  of  OT&E, 
pointing  out  that  the  existing  SIMNET-D/SE  is  “good  enough”  for  developing  maneuver 
tactics,  techniques,  procedures,  and  doctrine.  However,  there  are  many  limitations  within 
the  SIMNET-D  that  restrict  its  value  for  evaluating  system  capabilities.  The  Operational 
Evaluation  Command  recommends  a  series  of  improvements  and  tasks  in  order  to  fully 
exploit  the  inherent  value  of  using  synthetic  environments  using  SIMNET-D  technology. 


[Department  of  the  Navy  1993a] 

Department  of  the  Navy.  Data  Request  for  Issue  Paper:  Simulation  Support  to  Readiness. 
Memorandum  for  the  Director,  Force  Structure  Analysis  Division  (OSD  PA&E). 
Unpublished  memorandum.  USN,  Washington,  DC,  1993. 

No  abstract  available. 


[Department  of  the  Navy  1993b] 

USN.  Pacific  Fleet  and  Atlantic  Fleet  Tactical  Training  Manual  (TTM).  CINCPAC 
FLTINST/CINCLANTFLTINST  3501.1.  USN,  Washington,  DC,  1 993 . 

This  instruction,  referred  to  as  the  Tactical  Training  Manual  (TTM),  is  the  single  source 
for  tactical  training  policy  and  requirements  for  Pacific  Fleet  and  Atlantic  Fleet  ships, 
submarines,  aircraft  squadrons,  and  tactical  staffs.  TTM  is  to  be  used  by  all  groups 
planning  or  conducting  training  to  ensure  consistent  training  practices  throughout  the 
fleets.  The  TTM  consists  of  6  chapters  with  over  30  appendices  that  address  unit-specific 
training  requirements.  The  most  important  chapters  cover  the  Tactical  Training  Strategy, 
the  Training  Process,  and  Measurements  and  Certification.  The  Tactical  Training  Strategy 
chapter  provides  a  road  map  to  achieve  maximum  combat  readiness  and  interoperability 
between  the  Atlantic  and  Pacific  Fleets.  It  discusses  training  philosophy,  training 
responsibilities,  tactical  training  organization,  and  the  tactical  training  cycle  necessary  to 
achieve  that  objective.  The  Training  Process  chapter  contains  training  precepts  and 
training  program  guidelines  for  operational  staffs,  warfare  commanders,  and  individual 
units.  It  discusses  shore  and  sea  basic  training,  intermediate  training,  and  advanced 
training  flows.  The  chapter  on  Measurements  and  Certification  discusses  the  training 
assessment  and  feedback  cycle,  including  establishing  training  objectives,  scenario 
development,  performance  criteria  selection,  data  collection  plan  development, 
reconstruction  and  analysis,  and  performance  assessment  and  feedback.  The  TTM  was 
written  for  use  by  command  and  training  personnel  at  all  levels  to  ensure  a  consistent 
training  program.  The  philosophy  of  the  instruction  is  well  presented  for  all  users,  but  the 
most  important  information  is  provided  in  the  appendices  which  provide  specific  training 
cycle  information  for  each  ship  class  and  aircraft  squadron  type  as  well  as  operational 
staffs  and  warfare  commanders.  This  information  is  required  for  every  Commanding 
Officer  and  Immediate  Superior  in  Command  who  want  to  adequately  plan  their  unit’s 
training  cycle. 


[Department  of  the  Navy  1994a] 

Department  of  the  Navy.  Surface  Force  Training  Manual.  COMNAVSURFPACINST/ 
COMNAVSURFLANTINST  3502.2.  USN,  Washington,  DC,  1994. 

This  manual  provides  a  comprehensive  training  program  that  integrates  a  sequence  of 
individual,  team,  and  unit  training  evolution  in  all  areas  applicable  to  the  Naval  Surface 
Forces  in  the  Atlantic  and  Pacific  Fleets.  It  is  the  primary  directive  for  planning. 


BIB- 19 


scheduling,  and  executing  all  cyclic  and  repetitive  training  requirements  within  the  Naval 
Surface  Forces.  There  are  two  significant  areas  in  this  instruction  that  have  an  effect  on 
modeling  and  simulation:  Tactical  Training  Strategy  training  cycle  and  the  Exercise 
Equivalency  Program.  Chapter  2  details  each  phase  and  step  of  the  Tactical  Training 
Strategy  training  cycle.  This  is  the  same  training  cycle  specified  in  the  Tactical  Training 
Manual.  Chapter  2  also  describes  which  phases  and  steps  can  be  accomplished  as  shore- 
based  training.  The  other  significant  area  is  detailed  in  Appendix  C,  Exercise 
Equivalencies.  This  appendix  provides  a  matrix  of  those  exercises  approved  for  readiness 
reporting  under  the  type  commander’s  Exercise  Equivalency  Program.  The  Exercise 
Equivalency  Program  includes  only  scenarios  run  on  one’s  own  ship’s  systems,  whether 
generated  from  shore-based  and  mobile  (van)  scenario  generators  or  embedded,  on-board 
scenario  generators.  The  following  is  a  list  of  the  shore-based  approved  scenario 
generators:  TACDEW,  ENWGS,  CSTS,  Mobile  Combat  Systems  Trainer  Device  20B4, 
Mobile  Combat  Systems  Trainer  Device  20B5,  Radar  Video  Recorder  (RAVIR),  LAMPS 
I/III  Mobile  Team  Trainer  Unit  (LMTTU),  and  AN/SQR-17/17A  “Rooftop”  Transmitter, 
Device  14E12  (RFTOP).  This  manual  was  written  for  use  by  command  and  training 
personnel  at  all  levels  to  properly  plan,  schedule,  conduct,  and  document  training.  It 
provides  clear  cut  guidance  on  training  requirements  and  acceptable  training  alternatives. 
This  information  is  required  for  every  Commanding  Officer  and  Immediate  Superior  in 
Command  who  want  to  adequately  plan  their  unit’s  training  cycle. 


[Department  of  the  Navy  1994b] 

Department  of  the  Navy.  Threat  Training  Manual.  CINCPACFLTINST/CINCLANT- 
FLTINST  S3057.1A.  USN,  Washington,  DC,  1994. 

This  instruction  is  a  SECRET  supplement  to  the  Tactical  Training  Manual.  The  Threat 
Training  Manual  provides  standardized  tools  with  which  trainers  and  operators  can 
develop  widely  varying  threat  scenarios  according  to  their  needs.  The  Threat  Training 
Manual  is  designed  to  shift  the  focus  of  Fleet  Tactical  Training  from  the  defunct  Soviet 
Union  toward  a  broad  understanding  of  the  other  multi-form  threats  that  U.S.  Naval 
Forces  could  confront.  Complementary  to  the  Tactical  Trdning  Manual,  the  Threat 
Training  Manual  provides  a  toolbox  of  potential  threat  capabilities  and  tactics  that  can  be 
used  to  develop  scenarios.  These  threat  scenarios  will  then  be  used  to  support  training 
objectives  throughout  the  Interdeployment  Training  Cycle  that  prepares  units  for  overseas 
deployment. 


[DMSO  1995] 

Defense  Modeling  and  Simulation  Office.  Modeling  and  Simulation  (M&S)  Master  Plan. 
Office  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology),  Washington,  DC, 
January  1995. 

The  M&S  Master  Plan  implements  policy  outlined  in  DoD  Directive  5000.59,  “DoD 
Modeling  and  Simulation  (M&S)  Management.”  It  establishes  DoD-wide  M&S 
objectives;  provides  a  comprehensive  framework  for  the  programming  and  budgeting  of 
M&S  projects,  programs,  and  activities;  and  assigns  responsibilities  for  its  implementation. 


[DSMC  1994] 

Defense  Systems  Management  College.  Systems  Acquisition  Manager 's  Guide  for  the 
Use  of  Models  and  Simulations.  DSMC,  Ft.  Belvoir,  VA,  September  1994. 

This  guidebook  is  the  result  of  an  1 1-month  Military  Research  Fellowship  program 
sponsored  by  the  defense  systems  acquisition  community  and  funded  by  the  Defense 
Modeling  and  Simulation  Office.  It  provides  the  acquisition  community  information  on 
DoD  policy  regarding  modeling  and  simulation  (M&S)  capability  and  how  M&S  can  be 
applied  throughout  the  acquisition  cycle.  It  also  gives  Project  Managers  an  understanding 
of  dual-use  technologies,  faster  and  lower  manufacturing  costs,  and  complements 
Operational  Test  and  Evaluation  (OT&E).  This  guidebook,  developed  as  a  reference  for 
the  Project  Manager,  describes  M&S  policies,  types  of  M&S,  applications,  and  key 
technical  and  management  issues.  The  study  is  organized  into  a  preface,  eight  chapters, 
and  an  epilogue,  with  appendices  listing  detailed  DoD  M&S  sources.  Chapter  One 
introduces  the  study,  explaining  the  purpose,  methodology,  and  objectives.  Chapter  Two 
discusses  today’s  applications,  the  systems  acquisition  process,  and  the  requirements 
generation  process.  Chapter  Three  gives  a  detailed  description  of  DoD  M&S, 
organization,  and  policy.  Chapter  Four  outlines  the  classification  of  M&S,  including  live, 
virtual,  and  constructive  technologies.  Chapter  Five  addresses  the  acquisition  life  cycle, 
and  Chapter  Six  outlines  the  acquisitions  issues  in  the  M&S  community.  Chapter  Seven 
addresses  management  issues,  and  Chapter  Eight  looks  to  the  future  of  M&S.  The  study  is 
an  excellent  and  comprehensive  work  outlining  DoD’s  M&S  organization  and  policy.  It 
identifies  the  key  task  for  the  future,  which  is  defining  and  providing  an  infrastructure  to 
combine  individual  M&S  into  a  seamless  live,  virtual,  and  constructive  world.  The  study 
addresses  the  need  for  standards  for  interface  definition,  communication,  representation  of 
environment,  management,  security,  field  instrumentation,  and  performance  measurement. 
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While  not  an  objective  of  this  study,  a  more  comprehensive  look  at  user  requirements 
generation  and  addressing  the  merger  of  combat,  materiel,  and  training  developments 
models  might  have  been  discussed  in  more  detail.  The  cost  of  M&S  is  driving  both  the 
training  and  OT&E  communities  in  this  direction.  The  fidelity  issue  for  training  versus 
OT&E  must  be  addressed.  From  a  cost  and  proficiency  standpoint,  separate  M&S  for 
each  community  is  unacceptable. 


[Eisleyetal.  1990] 

M.  E.  Eisley,  J.  D.  Hagman,  R.  L.  Ashworth,  and  M.  P.  Viner.  Training  Effectiveness 
Evaluation  of  the  Squad  Engagement  Training  System  (SETS).  RR  1562.  USA  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  Alexandria,  VA,  1990. 

No  abstract  available. 


[Fish  1995a] 

D.  Fish.  Return  on  Investment  Calculations  for  Indoor  Simulation  Marksmanship 
Trainer  (ISMT).  Memorandum  to  M&S  Benefits  Task  Force.  Marine  Corps  Modeling 
and  Simulation  Management  Office  (MCMSMO),  Quantico,  VA,  1995. 

The  purpose  of  the  ISMT  is  to  enhance  the  combat  training  of  Marines,  and  not  just 
replace  live  fire  with  simulated  rounds.  The  commander  may  wish  to  consider  the  results 
of  a  judicious  application  of  the  ISMT  to  a  routine  Marine  Battle  Skill  Training  (MB  ST) 
task  (requalification).  The  offset  of  just  one  day  of  firing  results  in  projected  savings  of 
$1.2  million  in  the  first  year  of  the  program.  In  years  two  through  five,  projected  annual 
savings  are  greater  than  $2. 1  million.  Complete  recovery  of  acquisition  costs  is  projected 
before  the  end  of  year  four;  projected  cumulative  savings  at  the  end  of  year  five  exceed 
$3.2  million. 


[Fish  1995b] 

D.  Fish.  Return  on  Investment  Calculations  for  Emerald  Light:  Instrumenting  Individual 
Combat  Entities.  Memorandum  to  M&S  Benefits  Task  Force.  Marine  Corps  Modeling 
and  Simulation  Management  Office  (MCMSMO),  Quantico,  VA,  1995. 

The  Emerald  Light  project  potentially  offers  the  Marine  Corps  a  great  deal  of  bang  for  the 
buck.  The  project  proposes  to  instrument  portions  of  the  Marine  Corps  Air  Ground 
Combat  Center  (MCAGCC)  at  29  Palms,  CA.  In  addition  to  the  instrumentation 
infrastructure,  equipment  will  be  procured  for  200  player  units,  including  riflemen,  tanks. 


LAVs  (light  armored  vehicles),  and  trucks.  This  instrumented  training  range  will  enable 
improved  computer-aided  exercises  (CAX)  at  MCAGCC  as  well  as  linkage  to  the  Army’s 
National  Training  Center  at  Ft.  Irwin,  CA.  A  Rough  Order  of  Magnitude  analysis 
suggests  that  Emerald  Light  could  reduce  the  costs  of  a  CAX  from  $9.5M  to  $5.8M. 
With  roughly  10  CAX  per  year,  recovery  of  investment  would  be  rapid.  The  improvement 
in  the  quality  of  Marine  Corps  training  and  joint  Army-Marine  Corps  is  not  quantified. 


[Fish  1995c] 

D.  Fish.  Return  on  Investment  Calculations  for  Deployable  Forward  Observer-Modular 
Universal  Laser  Equipment  (DFO-MULE).  Memorandum  to  M&S  Benefits  Task  Force. 
Marine  Corps  Modeling  and  Simulation  Management  Office  (MCMSMO),  Quantico,  VA, 
1995. 

The  Deployable  Forward  Observer-Modular  Universal  Laser  Equipment  (DFO-MULE)  is 
a  deployable,  modular,  distributed  interactive  simulation  (DIS)  compliant,  PC-based 
system  that  provides  basic,  advanced  and  sustainment  training  for  Forward  Observers  and 
Forward  Air  Controllers.  Fire  support  operations  involving  Naval  Gunfire,  Artillery,  and 
Close  Air  Support  are  simulated  on  this  system.  Assuming  a  10%  offset  of  live  fire  tasks, 
savings  in  ammunition  expenditures  realized  by  DFO-MULE  can  potentially  recover  the 
acquisition  costs  before  the  end  of  the  second  program  year.  The  most  expensive  round, 
the  $36,087.68  Copperhead,  is  excluded  from  the  calculation.  Several  qualitative 
advantages  are  also  cited. 


[Frost  and  Sullivan  1989] 

Frost  and  Sullivan,  Inc.  The  US  Military  Trainer  and  Simulator  Market:  Vols.  I  and  II. 
January  1989. 

This  is  a  two-volume  comprehensive  report  based  on  studies  conducted  by  Frost  & 
Sullivan.  It  provides  detailed  information  regarding  trainers,  simulators,  and  associated 
markets.  While  trainers  and  simulators  may  play  a  vital  role  in  the  conduct  of  a  wargame 
model  simulation,  there  is  a  distinct  difference  between  trainers  and  simulators,  and 
wargames,  models,  and  simulations.  The  report  does  not  address  wargames  and 
simulations  and  therefore  should  not  be  used  as  a  source  document  to  evaluate  trends  or 
markets  in  that  area. 

Section  One  is  the  Executive  Summary  which  provides  a  forecast  of  the  US  military 
market  for  simulators  and  other  training  devices  (FY90  -  FY94),  a  list  of  the  top 


companies  in  this  market  (FY83  -  FY88),  and  a  high-level  summary  of  the  entire  report, 
Section  Two  introduces  the  report’s  overall  objectives,  definitions,  organization, 
coverage,  sources  of  data,  application,  and  scope.  Section  Three  gives  background 
information.  Sections  Four,  Five,  and  Six,  which  make  up  the  majority  of  the  report, 
discuss  in  both  quantitative  and  qualitative  detail  each  Service’s  simulators  and  training 
devices;  planned  procurement;  research,  development,  testing,  and  evaluation  (RDT&E); 
competition;  and  contracts  and  forecasts.  Section  Seven  summarizes  the  market  for 
training  devices  related  to  the  military  market  but  funded  by  other  agencies.  Section  Eight 
discusses  technological  trends.  Section  Nine  profiles  manufacturers  of  military  simulators 
and  other  training  devices.  A  useful  glossary  constitutes  the  Appendix. 


[GAO  1993] 

General  Accounting  Office.  Army  Training:  Commanders  Lack  Guidance  and  Training 
for  Effective  Use  of  Simulations.  GAO,  Washington,  DC,  1993. 

This  study  was  conducted  in  response  to  a  tasking  from  the  Committee  on  Armed 
Services,  House  of  Representatives,  which  stated,  in  part,  “The  Army  faces  many 
constraints  on  the  field  training  exercises  that  it  has  traditionally  used  to  prepare  its  forces 
for  wartime  missions.  Funding  for  the  ammunition,  fuel,  and  maintenance  required  for 
these  exercises  has  been  reduced,  and  environmental  concerns  restrict  the  use  of  ranges 
and  maneuver  areas.  In  response,  the  Army  has  turned  to  simulations  to  supplement  field 
training  exercises.”  The  report  noted  that  at  the  brigade  level  and  above,  simulations  can 
be  used  to  improve  the  decision-making  skills  of  senior  battle  officers  before  they 
command  units  in  large-scale  training  exercises,  and  that  at  the  lowest  level,  simulations 
can  be  used  to  develop  the  basic  skills  of  individual  soldiers.  Exemplary  simulations  cited 
in  the  report  were  COFT  (Conduct  of  Fire  Trainer),  used  on  tanks  and  Bradley  Fighting 
Vehicles;  MILES  (Multiple  Integrated  Laser  Engagement  System),  used  to  simulate  direct 
fire  weapons  from  rifles  to  tank  and  helicopter  gunnery  systems;  and  SIMNET  (Simulator 
Networking),  used  to  provide  crew-,  platoon-,  and  company-level  training.  The  report 
concluded  that  simulations  are  not  reaching  their  full  potential  because  commanders  lack 
adequate  guidance  and  training  in  their  effective  use. 


[GAO  1991] 

General  Accounting  Office.  Army  Training:  Computer  Simulations  Can  Improve 
Command  Training  in  Large-Scale  Exercises.  GAO,  Washington,  DC,  1991. 

This  study  was  motivated  by  concern  over  whether  simulation  provides  an  adequate 
alternative  to  traditional  large-scale  military  exercises.  The  GAO  analyzed  the  1990 
REFORGER  exercise  (Return  of  Forces  to  Germany)  in  which  extensive  use  was  made  of 
simulation,  fewer  U.S.  troops  participated,  and  emphasis  was  given  to  training  staffs  and 
leaders  at  higher  organizational  levels  rather  than  lower-level  units.  Training  for  higher- 
echelon  leaders,  such  as  brigade,  division,  and  corps  levels,  was  made  possible  by 
extensive  use  of  computer-assisted  simulation.  It  was  reported  that  a  consensus  exists 
among  many  military  officials  that  computer-simulated  exercises,  such  as  REFORGER, 
offer  the  potential  for  effective  training,  particularly  at  higher  organizational  levels  where 
the  focus  is  on  battle  planning  and  command  and  control.  Benefits  of  such  training  include 
an  emphasis  on  battle  planning,  staff  procedures,  and  command  and  control;  more  efficient 
use  of  training  time;  a  focus  on  higher  echelons  that  would  otherwise  be  cost  prohibitive; 
and  lessened  adverse  environmental  and  political  impacts.  The  1990  exercise  saved  more 
than  $4M  in  transportation  and  cargo  handling  costs  as  compared  to  costs  historically 
documented.  However,  the  study  noted  that  comparing  the  costs  of  traditional  and 
simulation-based  exercises  may  not  be  meaningful  because  of  differences  in  the  type  of 
training  provided. 


[Garcia  etal.  1994] 

A.  B.  Garcia,  R.  P.  Gocke,  N.  P.  Johnson.  Virtual  Prototyping:  Concept  to  Production. 
Defense  Systems  Management  College,  Ft.  Belvoir,  VA,  March  1994. 

This  study  represents  the  combined  efforts  of  three  military  Research  Fellows  participating 
in  an  1 1-month  Senior  Service  College  Research  Fellowship  program.  The  program  was 
designed  to  provide  a  forum  for  the  conduct  of  research  in  a  subject  of  vital  interest  to  the 
U.S.  defense  acquisition  community.  The  research  that  supports  this  study  was  conducted 
during  calendar  year  1993.  Simply  stated,  this  study  assesses  and  describes  the  current 
state  of  virtual  prototyping.  It  focuses  on  leveraging  virtual  reality  to  “build”  prototypes 
with  computers,  test  their  performance  in  a  synthetic  battlefield,  conduct  trade-oflf 
evaluations  between  existing,  modified,  and  new  systems,  all  before  the  actual 
construction  of  a  prototype.  The  study  is  organized  into  an  Executive  Summary  and  six 


chapters.  The  introduction,  Chapter  One,  explains  the  objective,  methodology, 
assumptions,  and  background  of  the  study.  Chapter  Two  explains  the  defense 
environment  and  the  role  of  virtual  prototypes.  Chapter  Three  addresses  the  spectrum  of 
synthetic  environments.  Chapter  Four  examines  the  application  of  synthetic  environments 
to  the  acquisition  process.  A  detailed  account  of  synthetic  applications  is  given  in  Chapter 
Five,  while  Chapter  Six  presents  the  study’s  summary  and  recommendations.  There  are 
useful  appendices  that  list  virtual  prototype  models  and  simulations,  the  points  of  contact 
for  each  one,  a  glossary,  and  a  bibliography. 

The  body  of  literature  on  virtual  prototyping  is  without  extensive  references  to  current 
applications  in  the  defense  industry:  not  because  it  is  not  being  used  but  because  each 
company  considers  virtual  prototyping  a  key  ingredient  in  its  ability  to  compete.  Thus, 
companies  jealously  protect  the  specifics  of  how  they  use  virtual  prototyping.  As  an 
example,  Boeing  developed  its  newest  commercial  aircraft,  the  Boeing  777,  without 
building  a  physical  mock-up.  The  digital  prototype  was  designed  and  tested,  then  went 
straight  to  the  design  floor  for  construction.  Boeing  will  not  share  its  “how  to”  use  of 
virtual  prototyping.  Throughout  the  text  there  are  2  tables,  71  figures,  and  10  color  plates 
that  make  this  otherwise  difficult  subject  more  understandable.  The  study’s  literature 
review  identified  more  than  500  articles  of  possible  relevance.  Ultimately,  150  of  these 
sources  were  actually  used  in  the  preparation  of  the  study  and  113  are  further  cited  in  the 
bibliography. 


[Gates  1987] 

S.  M.  Gates.  Appropriate  Mix  of  Live  Fire  and  Simulated  Fire  During  Training.  CNA 
CRM  87-116.  Center  for  Naval  Analyses,  Alexandria,  VA,  1987. 

Gates  presents  a  small-scale  review  of  studies  to  estimate  the  general  benefits  of  TADS  S 
and  live  fire  to  develop  weapon  system  operator  proficiency.  The  general  conclusions  are 
that  training  with  simulated  fire  can  develop  the  same  skills  as  live  fire,  an  increase  in 
simulator  practice  time  can  translate  into  higher  levels  of  operator  proficiency;,  and  only 
minimal  amounts  of  live  fire  may  be  required  to  effect  transfer  of  simulator  training.  As 
simulator  training  becomes  more  effective,  more  live  fire  practice  can  be  replaced  without 
degrading  operator  proficiency,  but  a  certain  amount  of  live  fire  will  probably  always  be 
recommended  to  document  the  training  transfer  and  convince  the  gunner  that  he  is,  in  fact, 
a  capable  weapon  system  operator  and  not  merely  an  expert  at  a  realistic  video  game. 


[Gorman  1990] 

P.  F.  Gorman.  The  Military  Value  of  Training.  IDA  Paper  P-2515.  Institute  for  Defense 
Analyses,  Alexandria,  VA,  1990. 

This  paper  was  prepared  in  response  to  a  NATO  inquiry  seeking  to  apply  operational 
research  to  the  contribution  of  military  training  and  the  overall  effectiveness  of  the  force. 
It  presents  the  training  vision  of  General  Paul  Gorman  (USA  Ret.),  a  former  commander 
of  the  U.S.  Army  8th  Infantry  Division.  At  a  time  when  requirements  for  readiness  were 
increasing  and  training  resources  were  decreasing,  he  was  among  the  first  to  introduce 
battle  simulation  to  train  and  evaluate  tactical  Army  units.  This  paper  is  of  value  to 
historic  researchers  of  the  early  years  of  simulations  or  those  looking  for  basic  simulations 
applicable  at  the  tactical  level  of  combat  unit  training.  General  Gorman,  considered  one  of 
the  “founding  fathers”  of  training  simulations,  has  written  an  easily  understandable  study. 

This  document  is  a  primer  on  the  use  of  training  simulations  at  the  tactical  level,  and  the 
author’s  architectural  framework  for  training  continues  to  be  followed  today.  General 
Gorman  writes,  “The  first  battle  of  most  wars  fought  by  the  Army  of  the  United  States 
was  a  disaster:  a  costly  defeat  or  a  Pyrrhic  victory.”  This  quotation  is  a  timeless 
argument  in  favor  of  simulated  training  that  is  resource  effective,  and  can  be  rehearsed 
repeatedly  to  improve  combat  readiness,  refine  necessary  skills  and  strategies,  and 
preserve  the  fighting  forces.  Gorman  presents  compelling  evidence  concerning  the  value 
of  combat  experience  for  survival  and  later  success  in  combat,  and  makes  the  case  that 
tactical  engagement  simulation  provides  the  military  with  the  equivalent  of  combat 
experience  without  its  casualties.  During  World  Wars  I  and  II,  inexperienced  pilots  had  at 
best  four  chances  out  of  ten  of  being  shot  down  and  fewer  than  15%  of  pilots  had  a  better 
than  50%  chance  of  surviving  their  first  combat.  But  with  each  successive  victory,  a 
pilot’s  survivability  increased  dramatically.  A  NATO  review  of  German  U-boat  captains 
showed  that  10%  of  them  accounted  for  45%  of  recorded  sinkings.  Early  success  in 
combat  was  correlated  strongly  with  survivability.  During  the  early  years  of  the  Vietnam 
war,  about  one  U.S.  fighter  was  lost  for  every  two  North  Vietnamese  MIGs;  the  ratio  of 
victories  to  losses  rose  to  12.5:1  for  Navy  pilots  after  the  Top  Gun  program,  an 
engagement  simulation,  was  established. 


[Grimes  1994] 

V.  P.  Grimes.  “Navy/Marine  Corps  Team  Changes  Training  Focus.”  National  Defense 
129  (502)  (November  1994):  26-27. 

This  article  details  simulation  system’s  marketing  trends  in  the  Navy  and  Marine  Corps.  It 
provides  readers  with  basic  knowledge  of  training  trends  in  the  Navy  Department 
triggered  by  changes  in  the  threat,  budget  constraints,  and  Joint  training  and  operational 
commitments.  There  are  enough  facts  and  figures  in  this  two-page  article  to  give  the 
reader  a  basic  knowledge  of  Navy  and  Marine  Corps  training  trends  as  well  as  reasons  for 
the  direction  training  has  taken  in  these  services.  It  covers  areas  such  as  Zero-based 
Training  and  Educational  Review,  systems  trainers,  littoral  warfare  environment; 
organizational  changes  are  given  in  just  enough  detail  to  provide  an  outline  of  the  naval 
Services’  training  initiatives. 


[Hammond  et  al.  1993] 

M.  H.  Hammond,  D.  R.  Graham,  and  E.  P.  Kerlin.  The  Role  of  Distributed  Simulation  in 
Defense  Acquisition.  IDA  Paper  P-2902.  Institute  for  Defense  Analyses,  Alexandria, 
VA,  1993. 

This  paper  examines  the  role  that  distributed  simulation  could  play  in  incorporating  the 
latest  design,  engineering,  manufacturing  and  support  technologies,  and  management 
practices  in  DoD’s  acquisition  system.  These  improved  technologies  and  practices  have 
allowed  some  of  the  most  progressive  commercial  firms  to  reduce  development  time  and 
costs  by  up  to  50  percent.  Three  advances  have  made  this  possible:  (a)  the  diffusion  of 
new  management  approaches  such  as  integrated  product  and  process  development  (IPPD) 
and  Total  Quality  Management  (TQM);  (b)  rapid  improvements  in  design  and 
manufacturing  technologies;  and  (c)  the  increased  power  of  distributed  communications 
networks  linking  dispersed  elements  of  design  teams.  Discussions  are  provided  on  how 
commercial  firms  and  defense  programs  have  used  distributed  simulation  technologies  in 
conjunction  with  these  advances  to  provide  a  basis  for  coupling  simulation  investments 
with  acquisition  process  improvements.  Specific  investment  areas  are  identified  which 
would  aid  the  DoD  and  its  suppliers  to  adopt  these  advances. 


[Hammond  et  al.  1995] 

M.  H.  Hammond,  B.  Brykczynski,  E.  A.  Feustel,  A.  N.  Joglekar,  J.  J.  Loughran.  Review 
of  Defense  Modeling  and  Simulation  Office  Projects  and  Activities  for  Fiscal  Years 
1992-1994.  DDA  Paper  P-3062.  Institute  for  Defense  Analyses,  Alexandria,  VA,  1995. 

This  paper  provides  a  consolidated  summary  of  those  projects  and  activities  sponsored  by 
the  Defense  Modeling  and  Simulation  Office  (DMSO)  during  Fiscal  Years  1992-1994. 
The  projects  were  selected  by  a  formal,  community-Avide  process  led  by  DMSO,  whereas 
the  activities  were  directly  selected  by  DMSO.  Representative  projects  and  activities 
range  from  development  of  simulation  protocols  to  participation  in  exercises  to 
coordination  of  a  DoD  directive  on  M&S  management.  A  total  of  48  projects  and  17 
activities  were  reviewed.  Summary  descriptions  for  each  project  and  activity  are 
presented,  including  such  information  as  the  objectives,  accomplishments,  and  benefits  of 
the  projects  and  activities.  Aggregated  information  relating  the  projects  and  activities  to 
common  themes  is  also  provided.  In  addition,  the  process  by  which  the  projects  were 
selected  is  also  described.  This  papers  thus  serves  as  a  reference  furnishing  summary 
information  on  the  DMSO  projects  and  activities.  Making  information  available  in  this 
way  should  help  other  parties  identify  results  of  the  projects  and  activities  that  could  be  of 
use  to  them. 


[Hawley  and  Christ,  in  press] 

J.  K.  Hawley  and  R.  E.  Christ.  Assessing  the  Value  Added  of  New  Training  Technologies. 
ARI  TR  (TBD).  USA  Research  Institute  for  the  Behavioral  and  Social  Sciences, 
Alexandria,  VA,  in  press. 

The  authors  present  a  value-added  assessment  model  for  Distributed  Interactive 
Simulation  (DIS)  training.  They  applied  portions  of  the  model  to  the  Multi-Service 
Distributed  Training  Testbed  (MDT2)  Plus.  This  testbed  is  designed  to  train  Close  Air 
Support  (CAS)  and  counter  air  defense  in  multi-Service  operations.  The  authors  define 
“value  added”  by  asking  four  questions:  1 .  Is  the  potential  application  suitable!  2.  Is  the 
application  elective!  3.  Is  the  potential  application  superior  to  conventional  training 
methods?  4.  Is  this  superiority  operationally  relevant!  They  detail  and  provide 
measurement  concepts  for  each  of  the  questions.  In  their  test  of  the  model,  they  address 
two  of  the  assessment  dimensions:  suitability  and  estimated  effectiveness.  They  conclude 
that  MDT2  Plus  is  suitable  for  its  intended  purpose  if  it  is  not  used  for  introductory  CAS 
training,  or  as  a  complete  substitute  for  field  training.  They  also  conclude  that  MDT2  Plus 
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is  potentially  effective  in  training  multi-Service  CAS  synchronization  and  coordination 
tasks. 


[Horowitz  et  al.  1992] 

g  Horowitz,  C.  Hammon,  P.  Lurie,  P.  Brooks,  and  A.  Rolfe.  Ths  Cost-Effectiveness  of 
Flying  Hours  and  Simulators.  Briefing.  Institute  for  Defense  Analyses,  Alexandria,  VA, 
1992. 

The  authors  gathered  and  analyzed  data  relating  to  flying  hours  and  simulator  time  and 
mission  performance  or  safety.  More  than  $10B  per  year  is  spent  on  flying  hours  but  the 
appropriate  mix  of  flying  time  and  simulator  time  is  unclear.  A  general  conclusion  from 
several  different  studies  is  that  both  flying  hours  and  simulator  time  affect  performance, 
but  usually  career  flying  hours  have  a  greater  effect  than  recent  flying  hours.  Bombing  and 
air  drop  accuracy  data  indicate  that  additional  simulator  hours  seem  to  have  a  greater 
effect  than  additional  flying  hours,  and  simulator  hours  cost  at  most  a  third  as  much. 
Helicopter  accident  data  indicate  that  both  flying  hours  and  simulator  hours  reduce 
accidents,  but  simulator  hours  do  not  increase  expose  to  risk.  The  authors  suggest  that  it 
may  be  cost  effective  to  substitute  small  amounts  of  additional  simulator  time  for  flying 
time,  but  note  that  measures  of  proficiency  used  in  analyses  were  incomplete  and  the 
authors  were  wary  of  carrying  this  too  far. 


[Houck  et  al.  1991] 

M.  R.  Houck,  G.  S.  Thomas,  and  H.  H.  Bell.  Training  Evaluation  of  the  F-15  Advanced 
Air  Combat  Simulation.  AL-TP- 199 1-0047.  Air  Force  Armstrong  Laboratory,  Williams 
AFB,  AZ,  1991. 

The  evaluation  investigated  the  utility  of  existing  multiship  simulation  for  training  air 
combat  tasks.  F-15  pilots  and  air  weapons  controllers  participated  in  four  days  of  training 
in  simulated  air  combat  missions  in  an  unrestricted  combat  environment  that  included 
multiple  air  and  ground  threats,  electronic  warfare,  and  real-time  kill  removal.  After 
training,  participants  rated  the  value  of  training.  Overall,  participants  indicated  that 
simulation  enhanced  their  combat  readiness  and  was  more  beneficial  in  some  areas  than 
traditional  unit  training. 
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[Hughes  et  al.  1988] 

C.  R.  Hughes,  W.  G.  Butler,  B.  S.  Sterling,  and  A.  W.  Berglund.  Ml  Unit  Conduct  of 
Fire  Trainer:  Post  Fielding  Training  Effectiveness  Analysis.  USA  TRAC-WSMR  TEA 
16087.  USA  TRADOC  Systems  Analysis  Activity,  White  Sands  Missile  Range,  NM, 
1988. 

This  analysis  evaluated  the  training  effectiveness  of  the  Ml  UCOFT.  The  evaluation 
included  369  tank  commander-gunner  pairs  from  6  armor  battalions  in  Europe.  Training 
effectiveness  was  evaluated  in  terms  of  crew  performance  on  Tank  Table  VIII.  Results 
were  (1)  progress  through  the  training  matrix  and,  to  some  extent,  more  training  time 
typically  resulted  in  improved  Table  VIII  performance;  (2)  the  UCOFT  was  not  a  precise 
predictor  of  performance;  (3)  the  UCOFT  could  make  a  substantial  contribution  to 
sustaining  crew  gunnery  skills;  and  (4)  users  felt  that  the  UCOFT  improved  home  station 
gunnery  training. 


[Hughes  1989] 

W.  P.  Hughes,  ed.  Military  Modeling.  Second  edition.  Military  Operations  Research 
Society,  Alexandria,  VA,  1989. 

This  textbook,  used  at  the  U.S.  Army  War  College  with  the  endorsement  of  the  Military 
Operations  Research  Society  (MORS),  describes  the  attributes  of  well-conceived  military 
models,  shows  how  models  can  contribute  to  the  decision  process,  and  cautions  the  reader 
that  models  can  be  misused  and  oversold.  It  is  designed  for  the  reader  who  is 
knowledgeable  in  military  operations  and  the  decision-making  process,  but  who  has  only 
superficial  acquaintance  with  the  models  that  inform  and  contribute  to  the  process.  For 
the  subject  matter  expert.  Military  Modeling  serves  as  the  primer  for  military  modeling. 
The  book  begins  with  an  Overview  written  specifically  for  the  novice  and  proceeds  to 
introduce  the  student  to  the  subject  of  modeling  with  general  readings  in  15  technically 
oriented  chapters.  This  work  describes  military  models  across  all  dimensions.  Designed 
and  written  for  senior  individuals  who  are  entering  the  field  of  military  modeling,  it  serves 
as  the  transition  piece  from  those  who  know  about  modeling  to  those  who  are  expert  in 
the  field.  The  contributing  authors  represent  leading  practitioners  of  military  modeling. 
Of  note  is  the  discussion  of  models’  verification  and  validation;  these  often  misunderstood 
concepts  are  examined  in  depth,  clarified,  and  defined.  For  example,  the  author  notes  that 
all  models  are,  by  definition,  invalid  because  they  are,  after  all,  abstractions  of  an  actual 
activity.  In  addition  to  abstraction,  which  is  the  omission  of  some  observable  factors. 
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there  is  another  concept  that  affects  models:  approximation.  Approximation  involves 
limits  on  the  precision  of  calculation.  The  better  the  modeler  is  able  to  reduce  the 
influences  of  abstraction  and  approximation,  the  better  the  model. 


[Irvine  1994] 

G.  M.  Irvine.  “Combat  Training  Can  Be  Done  Pierside.”  Naval  Institute  Proceedings 
120  (January  1994):  76-79.  U.S.  Naval  Institute,  Annapolis,  MD. 

This  article  describes  the  advantages  of  on-board  training  with  a  ship’s  own  hardware 
using  either  the  SQQ-89  Onboard  Trainer  (OBT)  or  an  Interface  Test  Set  (ITS)  to  provide 
training  inputs.  Highlighting  exercises  from  1992  and  1993,  the  article  describes  how 
these  exercises  demonstrated  the  effectiveness  and  efficiency  of  training  in  port  using 
simulation  and  training  devices  in  conjunction  with  a  ship’s  own  systems.  It  demonstrates 
the  uses  of  simulation  in  education  and  training  today,  and  is  a  good  source  of  examples 
which  would  lend  credence  in  the  classroom. 


[Institute  of  Simulation  and  Training  1995] 

Institute  of  Simulation  and  Training.  12th  DIS  Workshop  on  Standards  for  the 
Interoperability  of  Distributed  Simulations.  Volume  I:  Position  Papers.  1ST,  Orlando, 
FL,  1995. 

This  document  is  a  compendium  of  101  technical  papers  presented  at  the  12th  DIS 
Workshop.  It  represents  the  very  latest  thinking  for  current  distributed  interactive 
simulation  technology  and  innovation.  The  15-page  Table  of  Contents  lists  all  titles, 
authors’  names  and  organizations.  A  sampling  of  some  of  the  more  provocative  titles 
includes  “Strategies  for  Scaling  DIS  Exercises  Using  ATM  Networks,”  “Qualitative  and 
Quantitative  Comparison  of  Images  fi’om  Dissimilar  Image  Generators  in  Distributed 
Visual  Simulation,”  “DIS  Synthesis  with  Interactive  Television:  Revolution  of  the 
Educational  Paradigm,”  and  “An  Algorithm  for  Presenting  a  Continuous  After-Action 
Review  Representation  fi-om  Multiple  Overlapping  DIS  Exercise  Segments.” 


[ITEC  1995] 

International  Training  Equipment  Conference  and  Exhibition  Proceedings.  N.  Jackson 
and  N.  Cruz,  compilers.  ITEC,  Ltd.,  The  Hague,  April  25-27,  1995. 

This  work  contains  papers  and  reports  from  the  International  Training  Equipment 
Conference  and  Exhibition  Proceedings  held  in  The  Hague,  The  Netherlands,  April  25-27, 
1995.  Materials  were  submitted  under  the  following  general  headings; 


Training  Performance 

Driving  Simulation  - 
Technical 

Driving  Simulation  - 
General  and  Applications 

Unseen  Services  To 
Aviation 

S5mthetic  Environment 

Distributed  Interactive 
Simulation  Technologies 


Computer  Generated  Forces 
-  Environment 

Distributed  Interactive 
Simulation 

Audio-Visual  Cues 

Multimedia  Training 
Requirements 

Airline  Training 

Flight  Training  Organization 


Aviation  Training 

Virtual  and  Constructive 
Simulation 

Computer  Based  Training 
Applications 

Simulation  Training  and 
Equipment 

Combat  Training 

Maritime  Training 


[JCS  1993] 

Joint  Chiefs  of  Staff.  The  Catalog  of  Wargaming  and  Military  Simulation  Models.  12th 
edition.  U.S.  Department  of  Defense,  Washington,  DC,  1993. 

No  abstract  available. 


[Jolly  and  Ward  1995] 

A.  C.  Jolly  and  R.  C.  Ward.  Cost  Saving/ Avoidance  Benefits  of  Hardware-in-the-Loop 
Simulation  and  Distributed  Interactive  Simulation  at  RD&E  Center,  US  Army  Missile 
Command.  Research,  Development,  and  Engineering  Center,  USA  Missile  Command, 
Redstone  Arsenal,  AL,  1995. 

Jolly  and  Ward’s  report  contains  the  details  of  the  outcomes  of  selected  use  of  Hardware- 
in-the-Loop  (HWIL)  and  Distributed  Interactive  Simulation  (DIS).  They  provide  some 
examples  of  demonstrated  benefits  in  cost  savings  and  cost  avoidance  experienced  in 
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missile  development  programs  supported  by  the  Research,  Development,  and  Engineering 
Center  (RDEC). 


[Kemins  et  al,1990] 

J.  W.  Kemins,  N.  K.  Atwood,  and  J.  T.  Root.  Concept  for  a  Common  Performance 
Measuring  System  for  Unit  Training  at  the  National  Training  Center  (NTC)  and  with 
Simulation  Networking  (SIMNET).  ARI  Research  Report  1574.  USA  Research  Institute 
for  the  Behavioral  and  Social  Sciences,  Alexandria,  VA,  1990. 

This  report  presents  a  concept  for  the  design  of  a  common  performance  measurement 
system  for  units  training  at  the  National  Training  Center  (NTC)  and  units  training  with 
simulator  networking  (SIMNET).  The  measurement  system  included  components  from 
mission  accomplishment,  performance  of  critical  tasks  and  expert  judgment.  The 
approach  presented  in  this  research  is  applicable  when  designing  evaluations  to  compare 
different  training  systems. 


[Kraemer  and  Bessemer  1987] 

R.  E.  Kraemer  and  D.  W.  Bessemer.  US.  Tank  Platoon  Training  for  the  1987  Canadian 
Army  Trophy  (CAT)  Competition  Using  a  Simulation  Networking  (SIMNET)  System. 
Research  Report  1457.  USA  Research  Institute  for  the  Behavioral  and  Social  Sciences, 
Ft.  Knox,  KY,  1987. 

The  authors  estimate  the  effects  of  SIMNET  training  based  on  observation  and  interviews, 
results  of  CAT  competition,  potential  relationships  between  CAT  results,  and  SIMNET 
training  with  other  unit  training.  Findings  suggest  that  SIMNET  training  may  have  helped 
units  develop  and  improve  their  fire  control  distribution  plans  and  helped  unit  leaders 
develop  the  command,  control,  and  communications  skills  to  effectively  execute  those 
plans  during  platoon  battle  runs.  Other  major  contributing  factors  to  CAT  outcomes  were 
conducting  live  fire  battle  runs  and  tank  crew  gunnery  training  on  the  Ml  UCOFT  (Unit 
Conduct  of  Fire  Trainer).  The  most  apparent  shortcoming  of  the  SIMNET  that  may  have 
interfered  with  CAT  training  was  in  the  Ml’s  fire  control  system. 


[Lane  and  Alluisi  1992] 

N.  E.  Lane,  and  E.  A.  Alluisi.  Fidelity  and  Validity  in  Distributed  Interactive  Simulation: 
Questions  and  Answers.  IDA  Paper  D-1066.  Institute  for  Defense  Analyses,  Alexandria, 
VA,  1992. 

No  abstract  available. 


[McIntyre  and  Salas  1995] 

R.  M.  McIntyre  and  E.  Salas.  “Measuring  and  Managing  Team  Performance:  Lessons 
from  Complex  Environments.”  In  Team  Effectiveness  and  Decision  Making  in 
Organizations,  R.  A.  Guzzo  and  E.  Salas,  eds.  Jossey-Bass,  San  Francisco,  CA,  1995. 

The  authors  present  20  principles  concerning  team  training  derived  from  simulator-based 
research  of  military  teams.  These  were  Navy  tactical  decision-making  (TAC-DM)  teams 
which  make  decisions  under  the  stresses  of  time  pressure  and  enemy  threat.  Taskwork 
(tasks  performed  by  individual  team  members)  is  contrasted  to  teamwork  (team 
interactions  which  improve  the  quality  of  team  performance).  Principles  are  grouped  in 
terms  of  teamwork  behaviors,  team  leadership,  and  roles  of  individual  team  members. 


[Morrison  et  al.  1991a] 

J.  E.  Morrison,  E.  H.  Drucker,  and  D.  A.  Campshure.  Devices  and  Aids  For  Training  Ml 
Tank  Gunnery  in  the  Army  National  Guard:  A  Review  of  Military  Documents  and  the 
Research  Literature.  ARI  Research  Report  1586.  USA  Research  Institute  for  the 
Behavioral  and  Social  Sciences,  Alexandria,  VA,  1991. 

This  report  presents  a  review  of  military  documents  and  the  research  literature  on  the 
devices  and  aids  that  could  be  to  train  and  sustain  gunnery  skills  and  knowledge  for  use 
within  the  Army  National  Guard.  Six  gunnery  skills  training  devices  were  examined  on 
the  basis  of  skill  acquisition,  retention,  prediction  of  performance  and  transfer  of  training. 
The  beneficial  effects  of  partial  functional  overlaps  of  the  devices  were  noted  from  both  a 
learning  and  operational  perspective. 


[Morrison  et  all 99 lb] 

J.  E.  Morrison,  D.  A.  Campshure,  and  E.  L.  Doyle.  A  Device/Aid-Based  Strategy  for 
Training  MJ  Tank  Gunnery  in  the  Army  National  Guard.  ARI  Research  Report  1587. 
USA  Research  Institute  for  the  Behavioral  and  Social  Sciences,  Alexandria,  VA,  1991. 

Training  devices  and  aids  can  minimize  constraints  to  on-tank  gunnery  training  in  the 
Army  National  Guard  (ARNG).  This  report  describes  a  strategy  for  employing  devices 
and  aids  to  train  gunnery.  Although  specifically  tailored  to  the  needs  of  the  ARNG,  the 
discussion  of  methods  should  be  useful  to  the  design,  development,  and  implementation  of 
gunnery  training  in  the  Active  Component  and  the  U.S.  Army  Reserve. 


[Morrison  and  Hagman  1994] 

J.  E.  Morrison  and  J.  D.  Hagman.  A  Device-Based,  Time-Compressed  Strategy  for  Army 
National  Guard  Tank  Gunnery  Training.  ARI  Research  Report  1012.  USA  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  Alexandria,  VA,  1994. 

This  report  describes  a  training  strategy  to  reduce  or  compress  the  time  needed  to  prepare 
for  tank  crew  qualification  on  Table  VIII  through  use  of  the  Conduct-of-Fire  Trainer 
(COFT)  and  Guard  Unit  Armory  Device  Full-Crew  Interactive  Simulation  Trainer,  Armor 
(GUARDFIST  I).  To  compress  time,  the  authors  recommend  that  training  (a)  be  focused 
on  only  gunnery  skills  needed  for  Table  VIII  qualification,  (b)  be  given  only  to  crews  with 
demonstrated  performance  deficiency,  and  (c)  be  devoted  to  those  Table  VIII 
engagements  found  to  be  most  difficult.  Although  the  strategy  was  designed  specifically 
for  use  by  armor  units  of  the  U.S.  Army  National  Guard,  the  rationale  supporting  its 
development  could  be  applied  to  other  situations  where  the  type  and  availability  of 
operational  equipment  and  training  devices  differ. 


[Nance  etal.  1994] 

J.  F.  Nance,  R.  P.  Neisler,  G.  W.  Steadman,  and  B.  Wilhoite.  Simulation  Offset  to  Live 
Fire  Training:  Test  Plan  Phase  3.  CIM-344.  Center  for  Naval  Analyses,  Alexandria, 
VA,  1994. 

No  abstract  available. 


[National  Simulation  Center  1994] 

National  Simulation  Center.  Training  with  Simulations:  A  Handbook  for  Commanders 
and  Trainers.  Combined  Arms  Center,  Fort  Leavenworth,  KS,  October  1994. 

This  U.S.  Army  Command  and  General  Staff  College  (CGSC)  text  was  written  for 
commanders  and  trainers  to  assist  them  in  the  planning  and  conduct  of  exercises  involving 
simulations.  It  is  designed  to  provide  the  reader  with  a  basic  introduction  to  simulations 
and  insight  into  how  command  and  control  training  simulations  may  be  incorporated  into 
an  overall  training  strategy,  specific  guidance  for  their  use  in  the  development  of  a  unit 
training  strategy,  general  guidance  into  the  planning  of  training  exercises,  and  a  view  of 
future  prospects  in  the  area.  Training  with  Simulations  consists  of  15  chapters,  7 
appendices,  and  a  glossary  of  abbreviations  and  terms.  The  National  Simulation  Center 
(NSC)  prepared  this  book  to  meet  the  varied  intellectual  requirements  of  a  primary  and  a 
secondary  readership.  Training  with  Simulations  adequately  serves  as  the  primary 
textbook  for  a  CGSC  course  and  as  reference  material  for  non-training  personnel  engaged 
in  command  and  control  training  simulations  development.  Although  techmcal  material  is 
included  to  ensure  the  thoroughness  of  the  topics  presented,  it  would  be  wrong  to 
characterize  this  book  as  a  technical  document.  By  design,  NSC  assumed  the  readers 
would  possess  very  little  inherent  knowledge  of  the  general  subject  of  simulations.  The 
text  offers  a  basic  foundation  of  the  topic  in  straight-forward  language,  including  a 
measured  assortment  of  details  that  serve  to  clarify  rather  than  distract.  In  the  initial 
pages,  this  book  clarifies  a  common  confusion  concerning  the  differences  between 
simulations,  models,  simulators,  and  the  varying  types  of  models  and  simulations. 
Thereafter,  the  book  continues  to  be  a  helpful  training  aid.  The  book  is  well  organized  for 
those  new  to  the  field  and  who  seek  to  develop  a  strong  point  of  departure  for  further 
reading  or  course  work  in  simulations.  Part  I  establishes  a  foundation  in  simulations.  Part 
II  addresses  the  question,  “Why  do  we  have  simulations  in  the  training  environment?” 
Part  III  discusses  a  number  of  simulations  currently  available  in  the  Army  as  members  of 
the  “Family  of  Simulations.”  Part  IV  presents  the  relationship  between  basic  Army 
training  principles  and  the  employment  of  training  simulations  into  a  simulations  strategy. 
Part  IV  is  divided  into  two  major  chapters  that  discuss  developments  to  the  year  2012. 


[Neuberger  and  Shea  1995] 

Thomas  K.  Neuberger  and  Dennis  P.  Shea.  Applying  Synthetic  Environments  to 
Operational  Training:  A  Perspective  from  Kernel  Blitz  95.  CNA  CAB  95-58.  Center  for 
Naval  Analyses,  Alexandria,  VA,  September  1995. 

No  abstract  available. 


[Noble  and  Johnson  1991] 

J.  L.  Noble,  and  D.  R.  Johnson.  Close  Combat  Tactical  Trainer  (CCTT)  Cost  and 
Training  Effectiveness  Analysis.  TRAC-WSMR-CTEA-91-018.  USA  TRADOC 
Analysis  Command,  White  Sands  Missile  Range,  NM,  1991. 

This  Cost  and  Training  Effectiveness  Analysis  (CTEA)  addressed  the  training  capabilities 
and  cost  effectiveness  of  the  CCTT,  a  simulator  intended  to  train  armor  and  mechanized 
infantry  crewmen.  The  analysis  concluded  that  the  CCTT  has  the  potential  to  train  tasks 
relating  to  command,  control,  and  communications;  maneuver  and  navigation;  teamwork 
and  leadership;  and,  to  a  lesser  degree,  certain  procedures  related  to  gunnery,  target 
acquisition,  and  driving.  The  analysis  projected  that  the  CCTT,  when  fielded,  would  be 
cost  effective  and  that  its  life  cycle  costs  would  be  paid  back  fully  during  its  service  life 
when  acquired  at  either  battalion  or  company  level. 


[Null  et  al.  1993] 

C.  H.  Null,  and  J.  P.  Jenkins,  eds.  NASA  Virtual  Environment  Research,  Applications, 
and  Technology.  National  Aeronautics  and  Space  Administration,  Washington,  DC, 
October  1993. 

This  paper  describes  the  environmental  research  technology  and  its  applications  in  NASA 
Centers,  the  potential  roles  it  can  take  in  NASA,  and  a  strategic  plan  for  NASA  in  the  next 
five  years.  Virtual  environment  displays  are  interactive,  computer  graphics  based,  and 
head  referenced,  creating  the  illusion  the  user  is  remoted  to  another  location.  The  paper 
contains  a  four-page  Executive  Summary,  the  Strategic  Plan,  and  numerous  one-  or  two- 
page  listings  of  programs,  research,  and  applications  currently  in  use  at  NASA.  Each 
listing  provides  a  description  of  the  system  capability,  system  status  and  future  plans, 
system  architecture,  location,  point  of  contact,  electronic  mail  address,  voice  and  fax 
telephone  numbers,  and  the  offices  with  which  coordination  has  been  conducted.  NASA 
has  contributed  to  the  current  body  of  literature  Avith  this  paper.  It  represents  an  overview 
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of  the  entire  NASA  virtual  environmental  effort  and  affords  the  reader  a  single-source 
document  from  which  to  initiate  research.  The  simulations  described  in  this  white  paper 
are  sophisticated,  state-of-the-art  programs.  The  applications  are  not  only  current  but 
demonstrate  futuristic  overtones. 


[ODUSD(R)  1995a] 

Office  of  the  Deputy  Under  Secretary  of  Defense  for  Readiness.  Military  Manpower 
Training  Report:  FY 1996.  U.S.  Department  of  Defense,  Washington,  DC,  1995. 

No  abstract  available. 


[ODUSD(R)  1995b] 

Office  of  the  Deputy  Under  Secretary  of  Defense  for  Readiness.  Use  of  Simulation  in 
DoD  Training.  U.S.  Department  of  Defense,  Washington,  DC,  1995. 

This  study  was  conducted  in  response  to  a  U.S.  Congressional  request  to  DoD  to 
determine  the  capabilities  and  limitations  of  simulation  for  training,  its  effect  on  training, 
DoD  investments,  and  whether  cost  savings  were  possible  though  increased  use  of 
simulation.  Some  key  findings  are  as  follows.  Regarding  capabilities,  simulations  cost 
much  less  to  use  than  operational  equipment  and  can  result  in  significant  economies  in 
training;  enable  people  to  do  things  that  would  otherwise  be  impossible,  such  as  realistic 
simulations  of  combat  and  mission  rehearsals;  enhance  safety  because  they  can  be  used  to 
do  things  that  are  too  dangerous  to  practice  with  operational  equipment;  in  some  cases, 
provide  performance  data  and  better  feedback  to  users  than  the  systems  they  represent; 
overcome  the  practical  limitations  of  live  training,  such  as  presenting  a  large  force  in  a 
360-degree  battlefield;  and  help  overcome  environmental  restrictions,  which  limit  the 
ability  to  perform  certain  types  of  combat  training,  such  as  electronic  warfare  and  realistic 
weapons  employment  training.  Regarding  limitations,  the  report  indicated  that  simulations 
have  a  limited  capability  to  represent  the  complex  conditions  involved  in  tactical  aviation, 
such  as  the  acceleration  forces  resulting  from  high-speed  maneuvers  and  concomitant 
problems  associated  with  spatial-temporal  awareness;  can  cause  a  type  of  disorientation 
known  as  “simulator  sickness”  if  visual  and  motion  cues  are  not  carefully  synchronized; 
and  still  have  technical  hurdles  to  overcome  in  order  to  create  a  “seamless  synthetic 
environment”  capable  of  including  all  combat  forces  and  platforms.  Technical  challenges 
include  developing  a  common  architecture,  authoritative  representations  of  environments 
and  human  behavior,  and  the  full  spectrum  of  weapon  systems  simulation.  Simulation  has 


proved  to  be  both  effective  and  efficient  in  providing  training  for  a  wide  variety  of  tasks. 
The  next  generation  of  simulations  presents  opportunities  for  greater  cost  savings,  but 
continuous  and  systematic  feedback  on  the  cost,  effectiveness,  and  limitations  of  M&S 
technology  is  critical  in  developing  successful  strategic  policies,  plans,  and  investments 
strategies. 


[OTA  1994] 

Office  of  Technology  Assessment.  Virtual  Reality  and  Technologies  for  Combat 
Simulation.  U.S.  Congress,  Washington,  DC,  1994. 

This  analysis  reviewed  the  state  of  the  art  of  virtual  reality  (VR)  technology  for  combat 
simulation,  and  made  a  number  of  observations  on  its  current  and  potential  cost 
effectiveness  and  in  the  future.  Cost  effectiveness  is  increasing  as  integrated  circuits 
increase  in  density.  The  number  of  transistors  in  microprocessors  is  doubling  every  two 
years,  the  information  storage  capacity  of  memory  chips  is  increasing  similarly,  and  CD- 
ROMs  and  other  media  are  now  commonly  used  for  high-capacity  storage.  Significant 
improvements  are  being  made  in  several  technologies  supporting  VR  (e.g.,  flat  panel 
display  chips,  LCDs  (liquid  crystal  displays),  micromechanical  mirror  arrays,  parallel 
processing,  networking,  improved  software);  together  these  will  allow  new  simulators  to 
improve  their  speed  and  realism.  Newer,  high-density  chips  reduced  the  average  cost  of  a 
fiilly  immersive  VR  system  (with  head  tracking,  head-mounted  display,  and  three- 
dimensional  sound)  from  $100K  in  early  1991  to  about  $50K  in  1993.  It  is  forecast  that 
cost  will  continue  to  decline.  Similar  trends  are  apparent  in  other  computer-intensive 
simulators.  High-tech  simulation  technology  continues  to  become  increasingly  cost 
effective.  VR  challenges  today  include  developing  improved  high-density,  color  flat-panel 
displays;  fast  head  tracking;  wideband  networks  with  low  latency;  multilevel  network 
security;  automating  object  and  world  description  for  scene  generators;  and  simulating 
infantry  and  noncombatants. 


[Operational  Research  and  Analysis  Establishment  1990] 

Operational  Research  and  Analysis  Establishment.  Comparison  of  Conventional  and 
Simulator  Enhanced  Tank  Gunnery  Training  Methods.  Project  Report  523.  Operational 
Research  and  Analysis  Establishment,  Ottawa,  Canada,  1990. 

The  report  describes  a  series  of  war  games  performed  with  the  JANUS  simulation  model 
to  determine  how  training  with  Ml  UCOFT  (Unit  Conduct  of  Fire  Trainer)  would 
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influence  combat  effectiveness.  Tank  gunners  trained  with  UCOFT  fired  their  opening 
rounds  about  25%  faster  than  conventionally  trained  gunners.  Based  on  estimated  combat 
effectiveness,  UCOFT  training  appeared  to  be  of  little,  if  any,  benefit  for  LEOPARD  Cl 
tanks  when  opposing  heavily  armored  probable  threats.  When  the  model  used  LEOPARD 
II  or  Ml  A1  tanks,  UCOFT-trained  gunners  killed  significantly  more  opposing  tanks  than 
conventionally  trained  gunners.  The  difference  in  combat  effectiveness  is  the  result  of  the 
relatively  greater  firepower  and  survivability  of  the  LEOPARD  II  and  MlAl  as  compared 
to  LEOPARD  Cl. 


[Orlansky  1993] 

J.  Orlansky.  The  Battle  of  73  Easting  and  Ways  to  Future  Victories.  NATO  Panel  8 
Meeting  on  Training  Strategies  for  Networked  Simulation  and  Gaming,  NATO, 
November  1993. 

“73  Easting”  is  a  location  in  the  Iraqi  desert  where  the  U.S.  Second  Armored  Cavalry 
Regiment  fought  elements  of  the  Iraqi  Tawakalna  Division.  The  battle  occurred  on  the 
second  day  of  the  four-day  ground  war.  The  U.S.  troops  were  outnumbered  three  to  one 
but  destroyed  the  opposing  force,  a  heavy  brigade  in  deliberate  defense  of  an  area  the 
Iraqis  had  used  for  training  exercises.  In  February  1992,  some  of  the  men  who 
participated  in  the  battle  came  to  Washington  to  review  and  correct  the  record  of  the 
battle.  Two  retired  officers  asked  them  the  following  question:  “None  of  you  have  ever 
been  in  combat  before.  How  do  you  explain  your  great  success  in  your  first  battle?” 
Battle  participants  responded,  “Sir,  this  was  not  our  first  battle.  This  was  our  15th  battle! 
We  fought  three  wars  at  the  National  Training  Center  (NTC),  Fort  Irwin,  California,  four 
wars  at  the  Combat  Maneuver  Training  Center  (CMTC),  Hoenfels,  Germany;  and  a  lot  of 
other  simulations  like  SIMNET,  COFT,  and  BCTP.  Yes,  sir,  we  had  been  ‘shot  at’ 
before.  Many  times.  This  war  was  just  like  out  training.” 


[Orlansky  et  al.  1994] 

J.  Orlansky,  C.  J.  Dahlman,  C.  P.  Hammon,  J.  Metzko,  H.  L.  Taylor,  and  C.  Youngblut. 
The  Value  of  Simulation  for  Training.  IDA  Paper  P-2982.  Institute  for  Defense 
Analyses,  Alexandria,  VA,  1994. 

This  study  examines  the  utility  of  simulation  for  training  at  individual,  unit,  and  Joint  force 
levels.  The  cost  effectiveness  of  flight  and  maintenance  simulators  is  well  established,  but 
a  relatively  limited  amount  of  work  has  been  done  to  establish  the  cost  effectiveness  of 


simulation  for  more  advanced  individual  training  in  units,  for  collective  training  in  units,  or 
for  units  in  component  or  joint  training.  One  finding  is  that  simulators  are  cost  effective 
for  initial  flight  and  maintenance  training  in  institutions:  they  train  as  well  as  actual 
equipment  and  cost  less  to  procure  and  use.  This  finding  also  applies  to  computer-based 
instruction  as  compared  to  traditional  classroom  instruction. 


[Orlansky  and  String  1977] 

J.  Orlansky,  and  J.  String.  Cost-Effectiveness  of  Flight  Simulators  for  Military  Training. 
Volume  I:  Use  and  Effectiveness  of  Flight  Simulators.  IDA  Paper  P-1275.  Institute  for 
Defense  Analyses,  Alexandria,  VA,  1977. 

The  operating  cost  of  flight  simulators  is  estimated  to  be  between  5  to  20%  of  the  cost  of 
aircraft.  Many  studies  have  shown  that  skills  learned  in  flight  simulators  can  be  performed 
successfully  in  aircraft;  consequently,  the  use  of  simulators  for  training  can  reduce  flight 
time.  At  the  time  of  this  study,  the  cost  effectiveness  of  flight  simulators  had  been 
reported  in  only  a  few  studies  and  their  primary  use  was  in  undergraduate  flight  training. 
Their  greatest  potential  for  future  savings  is  in  transition  and  continuation  flight  training. 


[Orlansky  et  al.  1984] 

J.  Orlansky,  M.  I.  Knapp,  and  J.  String.  Operating  Costs  of  Aircraft  and  Flight 
Simulators.  IDA  Paper  P-1733.  Institute  for  Defense  Analyses,  Alexandria,  VA,  1984. 

This  paper  summarizes  data  comparing  the  operating  cost  of  flight  simulators  and  aircraft. 
The  median  cost  ratio  of  simulators  to  aircraft  was  8%. 


[Orlansky  and  String  1981] 

J.  Orlansky  and  J.  String.  Cost-Effectiveness  of  Maintenance  Simulators  for  Military 
Training.  IDA  Paper  P-1568.  Institute  for  Defense  Analyses,  Alexandria,  V A,  1981. 

This  study  found  that  maintenance  simulators  are  as  effective  for  training  as  actual 
equipment  trainers  when  measured  by  student  achievement  in  school.  Acquisition  cost  of 
simulators  is  typically  less  than  actual  equipment  trainers.  In  the  majority  of  cases 
examined,  the  cost  to  develop  and  fabricate  one  unit  was  less  than  60%  of  actual 
equipment;  the  cost  of  fabricating  a  second  unit  was  less  than  20%.  It  was  estimated  that 
the  acquisition  and  use  of  a  maintenance  simulator  over  a  15-year  period  would  cost  38% 
as  much  as  actual  equipment.  In  studies  where  time  to  train  was  reported,  simulators  took 
25  to  50%  less  time  than  actual  equipment. 
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[Oswalt  1993] 

J.  Oswalt.  “Simulation  and  Gaming.”  International  Journal  of  Theory,  Design  and 
Research  24  (2)  (June  1993). 

This  special  edition,  a  compendium  of  articles,  summarizes  recent  military  simulation  and 
gaming  applications,  trends  that  affect  their  use,  characterization  and  criteria  of  evaluation, 
and  factors  affecting  their  development  (technologies,  standards,  and  requirements).  It 
also  describes  current  organizations  and  implementations  within  the  military  services, 
academic  institutions,  and  commercial  enterprises.  It  concludes  with  a  discussion  of  issues 
in  combat  simulation.  The  first  article,  “Current  Applications,  Trends,  and  Organizations 
in  US  Military  Simulation  and  Gaming,”  is  a  review  of  military  simulation  and  gaming. 
The  next  two  articles,  “Principles  for  the  Design  and  Selection  of  Combat  Simulations” 
and  “Flexible  Combat  Modeling,”  are  much  more  focused.  “Flexible  Combat  Modeling”  is 
of  particular  interest  to  the  DMSO  study.  The  author  not  only  discusses  the  contemporary 
period  but  postulates  about  what  will  constitute  necessary  flexibility  in  the  future.  He  also 
includes  a  brief  section  at  the  end  entitled,  “Advancing  the  State  of  the  Art.”  Here  he 
suggests  two  areas  for  research  and  design.  The  first  is  the  ability  to  represent  the 
differences  among  theaters  of  operation  and  forces.  This  effort  requires  defining  how 
conflict  would  likely  vary  by  region  and  then  determining  how  models  would  need  to  be 
adjusted  to  reflect  such  differences.  A  consistent  set  of  assumptions  is  yet  to  be  designed 
in  this  area.  The  second  area  is  the  representation  of  new  technologies.  The  author 
argues  that  new  technologies  or  threats  have  been  entered  into  models  without  adjusting 
the  strategy  or  operations  of  either  side,  leading  to  significant  but  questionable  effects  on 
the  overall  combat  outcomes.  Overall,  a  systematic  assessment  of  such  changes  in  combat 
environment  has  not  been  performed  by  the  military  analysis  community.  These  articles 
are  reprinted  from  earlier  publications. 


[OUSDAT  1994] 

Office  of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology.  Task  Force  on 
the  Use  of  Modeling  and  Simulation  in  Test  and  Evaluation.  U.S.  Department  of  Defense, 
Washington,  DC,  June  23,  1994. 

This  report  responds  to  the  Defense  Authorization  Report  for  FY  94  regarding  the  use  of 
modeling  and  simulation  in  test  and  evaluation.  The  response  is  structured  in  two  sections 
consisting  of  (1)  a  description  of  the  DoD  policy  regarding  modeling  and  simulation 
investment  and  a  progress  report  on  implementation,  and  (2)  a  discussion  of  the  analytical 
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basis  the  DoD  and  the  Services  have  used  as  an  investment  strategy.  Appendix  A  reflects 
where  modeling  and  simulation  has  realized  cost  savings  or  cost  avoidance. 


[Randel  et  al.  1992] 

J.  M.  Randel,  B.  Morris,  C.  D.  Wetzel,  and  B.  V.  Whitehill.  “The  Effectiveness  of  Games 
for  Educational  Purposes.”  Simulation  and  Gaming  23  (3)  (1992):  261-276. 

No  abstract  available. 


[Regianetal.  1992] 

J.  W.  Regian,  W.  Shebilske,  and  J.  Monk.  “A  Preliminary  Empirical  Evaluation  of  Virtual 
Reality  as  Instructional  Medium  for  Visual-Spatial  Tasks.”  Journal  of  Communication  42 
(4)  (1992):  136-149. 

No  abstract  available. 

[Pfeiffer  and  Dwyer  1991] 

M.  G.  Pfeiffer  and  D.  J.  Dvsyer.  Training  Effectiveness  of  the  F/A-18  Weapon  Tactics 
Trainer  (Device  2E7).  NTSC  TR  91-1008.  Naval  Training  Systems  Center,  Orlando,  FL, 
1991. 

This  study  is  a  training  evaluation  of  Device  2E7,  F/A-18  weapon  tactics  trainer.  Four 
maneuvers  were  examined,  two  each  involving  air-to-air  and  air-to-ground  weapons 
delivery  and  tactics.  Device  2E7  was  effective  for  achieving  learning  objectives;  skill 
acquired  on  the  device  transferred  to  the  F/A-18  aircraft  for  each  of  the  four  tasks 
examined. 


[Schendel  et  al.  1984] 

J.  D.  Schendel,  F.  H.  Heller,  D.  L.  Finley,  and  J.  K.  Hawley.  Use  of  Weaponeer 
Marksmanship  Trainer  in  Predicting  Ml 6A I  Rifle  Qualification  Performance.  RR  1370. 
USA  Research  Institute  for  the  Behavioral  and  Social  Sciences,  Ft.  Benning,  GA,  1984. 

No  abstract  available. 
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[Shlecter  and  Bessemer  1991] 

T.  M.  Shlechter  and  D.  W.  Bessemer.  The  Effects  of  SIMNET  Role-Playing  on  the 
Training  of  Prospective  Platoon  Leaders.  ARI  Technical  Report  938.  USA  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  Alexandria,  VA,  1991. 

This  investigation  examines  the  relative  value  of  having  prospective  armor  platoon  leaders 
engage  in  role-playing  activities  while  using  the  simulator  networking  (SIMNET)  training 
simulator.  The  results  demonstrate  that  being  in  a  platoon  leadership  position  during 
SIMNET  training  leads  to  the  highest  performance  evaluations.  These  findings  imply  that 
students  should  have  direct  experience  in  platoon  leadership  positions  during  their 
SIMNET  training. 


[Schwab  and  Gound  1986] 

J.  R.  Schwab  and  D.  Gound.  Concept  Evaluation  of  Simulation  Networking  (SIMNET). 
TR  86-CEP345.  USA  Armor  and  Engineer  Board,  Ft.  Knox,  KY,  1986. 

This  early  study  of  SIMNET  was  conducted  to  evaluate  its  capability  to  support  platoon 
level  command  and  control  exercises  to  train  individual  and  collective  tasks.  Three  of  the 
four  platoons  in  each  of  the  two  groups,  SIMNET  and  baseline,  improved  their 
performance  between  the  first  and  second  set  of  STXs  (simulation  training  exercises).  The 
SIMNET  group  improved  from  an  average  of  73%  GOs  on  the  pre-training  STX  to  an 
average  of  84%  on  the  post-training  STX.  The  baseline  group  improved  from  59  to  65% 
pre-  to  post-training.  The  SIMNET  group  had  a  higher  average  score  after  both  pre-  and 
post-training  STXs.  The  SIMNET  group  improved  its  average  group  score  by  13%  while 
the  baseline  group  improved  its  score  by  6%.  Test  players  expressed  the  opinion  that 
SIMNET  was  useful  at  training  platoons  in  troop  leading  procedures,  conunand  and 
control,  land  navigation,  and  teamwork.  They  cited  the  ability  to  place  platoons  being 
trained  throughout  an  exercise  as  one  of  the  advantages  of  the  system. 


[Seidel  and  Chatelier  1993] 

R.  J.  Seidel  and  P.  R.  Chatelier,  eds.  Advanced  Technologies  Applied  to  Training  Design. 
Plenum  Press,  New  York,  NY,  1993. 

This  is  a  collection  of  papers  that  is  the  product  of  a  workshop  sponsored  by  NATO’s 
Defense  Research  Group  Panel  8.  The  Group’s  overall  purpose  is  to  stimulate  the  defense 
application  of  research  through  the  active  exchange  of  information  among  the  NATO 


member  nations.  Panel  8  identifies  training  technologies  that  will  maintain  a  capable  and 
ready  force  during  periods  of  reductions  in  military  force  structure  and  budgets.  The 
workshop  participants  seek  to  leverage  opportunities  to  apply  these  training  technologies 
to  nonmilitary  roles  of  the  future.  The  workshop  focused  on  six  technological  areas. 
Authoring  Systems;  Models;  Embedded  Training  and  Simulation;  Advanced  Hardware 
Technology;  Use  of  the  Cognitive  Approaches;  and  Expert  Systems.  The  workshop  also 
considered  existing  data  structures  that  can  provide  both  researchers  and  practitioners 
with  ready  sources  of  information.  Authoring  Systems  is  a  high-cost  driver  for  the  most 
important  factor  of  effective  education  and  training.  The  workshop  examined  computer- 
aided  techniques  that  offer  alternatives  to  overcome  this  dilemma.  The  technological 
revolution  in  training  is  addressed  in  the  next  three  sections.  Models  offer  various 
solutions  to  reduce  error  from  variations  in  approach,  offer  standardized  formats,  and 
provide  interoperability  for  training  and  education.  Embedded  Training  and  Simulation 
offer  the  training  managers  alternatives  to  purchasing  expensive  military  equipment. 
Advanced  Hardware  Technology  is  the  critical,  often  limiting  factor  that  this  workshop 
addresses.  Experience  has  shown  that  software  advances  often  outpace  hardware 
advances.  The  focus  on  Cognitive  Approaches  helps  the  participants  understand  the  latest 
theoretical  underpinnings  to  improve  and  further  design  improvements.  The  last  section. 
Expert  Systems,  focuses  more  on  application  and  use  than  on  theory;  the  root  of  all  expert 
systems  is  theory  and  it  is,  therefore,  outside  the  scope  of  this  work  s  examination.  Each 
of  the  papers  is  a  technical  monograph  addressing  a  subject  of  very  limited  scope.  The 
introduction  assists  the  reader  put  into  perspective  the  intended  outcomes  from  the 
collection  of  papers,  which  is  the  exchange  of  information  on  advanced  training 
technologies  and  products  that  apply  to  NATO. 


[Signal  1993] 

“Modeling  Processes  Offer  Command  and  Control  Aids.”  Signal  (September  1993). 

The  author  suggests  new  modeling  methods  may  permit  analysts  to  assess  command  and 
control  capabilities  in  the  same  manner  that  combat  forces  are  simulated.  He  argues  that 
functional  process  improvements  that  can  view  all  aspects  of  command  and  control  will 
result  in  determining  optimum  use  and  benefit.  This  is  a  very  technical  article.  The  author 
quickly  develops  the  details  of  functional  process  improvements  on  existing  simulations 
and  artificial  languages.  His  recommendations  are  very  specific,  presenting  a  series  of 
recommendations  that  will  improve  command  and  control  capabilities.  The  application  of 


these  new  technologies  to  training  command  and  control,  as  well  as  improving  command 
and  control  in  simulations  and  wargaming,  is  apparent. 


[Simpson  et  al.  1995] 

H.  Simpson,  W.  D.  West,  and  D.  Gleisner.  The  Use  of  Simulation  in  Military  Training: 
Value,  Investment,  and  Potential.  DMDC  TR  95-007.  Defense  Manpower  Data  Center, 
Monterey,  CA,  1995. 

This  study  was  conducted  in  response  to  a  U.S.  Congressional  request  to  DoD  to 
determine  the  capabilities  and  limitations  of  simulation  for  training,  its  effect  on  training, 
DoD  investments,  and  whether  cost  savings  were  possible  though  increased  use  of 
simulation.  It  is  an  expanded  version  of  the  DUSD  (R)  (1995)  report.  Key  findings 
include  (1)  simulation  technology  is  advancing;  (2)  the  Services  have  accepted  and  are 
using  simulation  and  are  making  significant  investments  in  them;  and  (3)  simulations  have 
proved  to  be  cost-effective  alternatives  to  traditional  methods  of  training  in  many  areas. 
Regarding  the  potential  for  increased  use  of  simulation,  the  report  drew  the  following 
conclusions. 

Live  simulation  plays  a  vital  role  in  preparing  our  forces  for  combat.  A  case  can  be  made 
for  increasing  the  amount  of  live  simulation  in  the  inter-Service  and  Joint  area.  Increased 
cooperation  among  the  Services  in  this  area  has  potential  to  improve  the  Joint  training 
readiness  of  the  force.  Stand-alone  single-system  simulation  is  used  heavily  by  the 
Services  and  is  cost  effective  in  many  applications.  The  area  with  the  greatest  potential 
payoff  for  more  application  is  probably  in  combat  air  crew  training.  With  real  aircraft,  this 
type  of  training  poses  the  greatest  risk  for  loss  of  life  and  aircraft  if  mistakes  are  made. 
There  is  not  adequate  data  to  make  a  recommendation  regarding  the  increased  use  of 
simulators  in  lieu  of  OPTEMPO  (operating  tempo)  flying  hours  per  aircraft,  steaming  days 
per  ship,  or  ground  vehicle  miles.  In  general,  the  authors  believe  that  the  Services  should 
be  permitted  to  make  this  tradeoff  on  the  basis  of  their  assessment  of  the  effect  of 
simulation  on  the  requirement  to  maintain  readiness.  Virtual  simulation  has  the  potential  to 
enable  Joint  and  inter-Service  training  in  mission  areas  not  being  trained  in  sufficiently 
now  (e.g.,  close  air  support).  The  technology  permits  coordinated  training  among  the 
Services  while  individual  Service  elements  remain  at  their  home  stations.  This  technology 
is  new,  though  it  appears  to  have  great  potential.  Continued  investment  should  be  made  in 
this  area  to  develop,  test,  and  refine  the  technologies  to  enable  it  to  reach  its  full  potential. 
The  level  of  investment  in  constructive  simulation  is  relatively  modest  in  comparison  with 
the  other  types  of  simulation.  This  does  not,  however,  decrease  its  importance,  especially 


in  the  command  staff  and  Joint  training  areas.  The  Joint  Staff  and  the  various  Unified  and 
Specified  Commands  place  increasing  importance  on  the  use  of  constructive  simulations  in 
their  training  programs.  The  authors  believe  that  continued  investment  in  constructive 
simulation  based  training  is  essential  to  the  Joint  readiness  of  the  Total  Force. 


[Spindler  1989] 

G.  B.  Spindler.  AC/243  (Panel  7/RSG.  15)D/4  on  the  Military  Value  and  Cost 
Effectiveness  of  Training.  NATO,  Brussels,  Belgium,  1989. 

This  document  contains  a  number  of  case  studies  concerning  the  military  value  and  cost 
effectiveness  of  training. 


[Swinsick  1995] 

S.  R.  Swinsick.  Longbow  Apache  FCRRMAP  Model  Benefits.  RFI  response.  McDonnell 
Douglas  Helicopter  Systems,  Mesa,  AZ,  1995. 

The  benefits  associated  with  the  use  of  manned  simulation  in  Force  Development  Test  and 
Experimentation  (FDT&E),  shown  in  Table  BIB-1,  are  numerous  and  quantified.  FDT&E 
provided  excellent  points  of  comparison  since  the  scenarios  and  activities  conducted  in  the 
simulation  and  the  field  test  were  so  similar.  The  FDT&E  Phase  II  was  about  six  times  the 
cost  of  FDT&E  Phase  I.  It  required  far  more  equipment,  24  times  the  personnel,  took 
longer  per  trial  to  complete,  and  had  the  inherent  risks  associated  with  the  operation  of 


aircraft  in  a  realistic  environment. 

Table  BIB-1.  Longbow  FDT&E 


Resources 

Phase  I 

Manned  Simulation 

Phase  II 

Field  Test 

Cost  (O&M  Army) 

$712,000 

4,049,000 

Equipment 

1  Simulator 

4  AH-64D 

2  UH-60 

14  Ml  Tank 

10  M3  Fighting  Vehicle 

2  2S6 

20+ADU 

47+  Assorted  Vehicle 

Personnel  (Government) 

27 

663 

Mission  Turn-Around  Time 

2  hours 

6  hours 

Data  Reduction  Time 

4  hours 

80  hours 

Number  of  Trials 

32 

16 

Test  Period 

4  weeks 

6  weeks 

Safety 

No  Risk 

Moderate  Risk 
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FDT&E  with  the  Longbow  Integrated  Training  System  has  been  the  most  comprehensive 
Longbow  simulation  effort  to  date.  From  individual  training  through  record  trials,  many 
lessons  were  learned  and  quantified.  The  simulation  provided  the  training  necessary  to 
make  the  crews  proficient  with  the  Longbow  systems.  During  FDT&E,  the  crews  were 
extremely  competent  and  knowledgeable  in  the  use  of  the  Longbow  systems  and  were  able 
to  rapidly  correct  when  something  went  wrong. 

A  longer  collective  training  period  is  recommended.  The  team  standard  operating 
procedure  (SOP)  must  be  second  nature  and  crews  must  be  working  cohesively  as  a  team 
before  record  trials  begin.  The  crews  conducted  only  three  days  of  combat  missions  prior 
to  record  trials.  At  the  end  of  the  first  week  of  record  trials,  the  crews  had  reduced  threat 
hits  by  33%  and  by  the  end  of  week  two  the  threat  hits  dropped  another  30%.  Moving 
more  combat  missions  into  the  collective  training  phase  would  improve  performance  at  the 
start  of  record  trials. 

The  record  trials  identified  some  requirements  which  must  be  considered.  The  ability  to 
rapidly  modify  scenarios  or  add  new  systems  to  the  simulation  is  a  necessity.  Diversity  in 
threat  array  composition  and  force  movement  prevents  crew  complacency  with  the 
missions.  The  threat  force  must  be  located  to  provide  the  attack  helicopter  company 
maximum  maneuver  space  within  the  simulation  database  and  to  allow  greater 
experimentation  with  tactics,  techniques,  and  procedures  (TTP).  The  simulation  arena 
must  remain  as  flexible  and  changing  as  the  global  environment  around  us. 

As  experienced  in  FDT&E,  the  simulator  provides  benefits  in  many  ways.  New  tactics  can 
be  exercised  against  multiple  situations  with  no  risk  to  life  or  limb.  The  simulator  allows 
the  attack  helicopter  company  to  improve  proficiency  and  increase  safety  before  reaching 
the  field  by  establishing  and  proving  SOP  and  TTP. 

Simulation  supports  the  restructured  Army’s  size,  development  objectives,  and  acquisition 
process.  It  requires  less  operational  and  support  costs,  no  wear  on  active  equipment, 
fewer  test  personnel,  and  no  risk  to  life  either  human  or  machine.  FDT&E  has  provided 
the  quantifiable  arguments  for  the  application  of  simulation  to  training,  test,  and 
experimentation. 


[TECOM  1995] 

USA  Test  and  Evaluation  Command  (TECOM/  Virtual  Proving  Grounds  (VPG) 
Modeling  and  Simulation  Success  Stories.  Unpublished  manuscript.  TECOM,  Aberdeen, 
MD,  August  1995. 

Firing  Impulse  Simulator  (FIS).  The  FIS  at  Aberdeen  Test  Center  (ATC)  simulates  the 
recoil  and  trunnion  loads  and  ballistic  shock  effects  of  firing  impulses  for  tanks,  and  towed 
and  self-propelled  howitzers.  The  FIS  delivers  approximately  3  million  pounds  to  a 
system  to  fully  replicate  an  actual  firing  without  the  use  of  ammunition.  The  FIS  project 
was  a  $6.9M  investment  in  simulation  technology.  Simulated  firing  with  the  FIS  was 
performed  on  the  M1A2  trunnion  test  program  in  May  1994.  Simulated  firing  with  the 
FIS  saved  about  $23M  of  cost  avoidance  for  a  typical  trunnion  bearing  test.  Test  crew 
was  reduced  from  13  to  4  members.  In  addition,  nonammunition  firings  resulted  in 
significant  time  savings  and  significant  environmental  savings,  reducing  such  effects  as 
noise,  toxic  fumes,  and  blast  overpressures.  A  one-year  Trunmon  Bearing  Test  is 
presently  half  completed  to  support  the  M1A2  Abrams  and  Armored  Gun  System  (AGS). 
Future  tests  include  the  Ml  A1  Recoil  Proofing  Test  and  Composite  Vehicle  test. 

M830E1  Fuse  Testing.  In  support  of  the  M830E1  test  program,  Yuma  Proving  Ground 
(YPG)  developed  a  computer-based  virtual  test  range  simulation  for  evaluating  tank 
versus  helicopter  engagements.  Flight  paths  of  the  helicopter  and  trajectories  of  the 
protot5^e  round  were  modeled  based  on  actual  flights  and  previous  firings.  Simulated 
helicopter  engagements  were  conducted  with  an  actual  manned  tank,  with  projectile 
(simulated)  miss  distance  recorded  for  each  engagement.  Eighty  such  “virtual”  firings 
were  conducted.  Then,  the  simulated  engagements  were  verified  by  an  actual  engagement 
using  a  live  round  against  a  drone  aircraft.  The  actual  test  cost  was  $260,000.  This 
results  in  a  cost  avoidance  of  $1.5  M  for  not  having  to  fire  a  complete  test  of  live  rounds 
against  live  targets.  Although  no  further  testing  is  currently  scheduled,  this  technique  can 
be  enhanced  for  future  large-caliber  Air  Defense  applications. 

Moving  Target  Simulator  (MTS).  The  MTS  test  facility  at  ATC  is  designed  to  assess 
the  ability  of  the  M1A2  tank  crew  to  track  and  simulate  firing  on  images  of  simulated 
maneuvering  targets.  The  MTS  allows  immersion  of  complete  weapon  systems  (aircraft 
or  ground  based)  into  a  moving  visual  target  environment,  and  to  investigate  pointing  and 
tracking  performance  of  fire  control  systems.  The  simulator  projects  a  laser  spot  target  on 
the  inside  of  a  100-foot  hemispherical  dome.  With  the  test  vehicle  in  the  middle  of  the 
dome,  evasive  targets  can  be  simulated  for  many  different  ranges.  Use  of  the  MTS  is 
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estimated  to  save  $1.5M  per  year  compared  to  field  testing.  Approximately  500  simulated 
engagements  were  completed  in  about  3  weeks  during  a  test  scenario,  whereas  in  a  live 
engagement  it  would  have  required  two  weeks  just  to  set  up  and  fire  a  single  live  round. 
The  effect  on  acquisition  time  can  be  less  time  spent  in  testing  and  retesting.  The  MTS  is 
now  being  refurbished  for  conducting  gun  positioning,  weapon  pointing,  and  gunner 
tracking  testing  for  the  Ml  A2,  AGS,  and  120mm  mortar. 

Millimeter-wave  Simulation/Test  Acceptance  Facility  (STAF).  The  STAF  is  a  joint 
effort  between  the  Redstone  Technical  Test  Center  (RTTC)  and  the  U.S.  Army  Missile 
Command.  It  is  a  hardware-in-the-loop  simulator  for  testing  millimeter  wave  radar-guided 
missiles.  The  STAF  provides  testing  of  a  fully  assembled  “live”  missile  with  multiple 
computer-based  test  scenarios  such  as  targets,  ranges,  and  temperatures.  The  STAF 
allows  a  random  selection  of  production  munitions  rounds  from  a  Fly-to-Buy  lot  and  tests 
these  rounds  in  a  real-time,  non-destructive  simulation.  RTTC  conducted  a  detailed  cost 
tradeoff  for  the  Longbow  missile  program.  Some  of  the  assumptions  that  entered  into  the 
analysis  were  that  destructive  firing  of  four  missiles  a  month  would  be  replaced  by  STAF 
method  of  testing  six  missiles  per  month  by  simulation,  and  destructive  firing  of  four 
missiles  per  year  would  be  used  as  an  additional  confidence  builder  and  feedback  to  the 
model.  If  destructive  test  programs  were  performed  on  the  Longbow,  the  yearly  test  cost 
would  be  $12.5M.  The  STAF  test  by  simulation  method,  however,  only  costs  $1.8[M] 
per  year,  resulting  in  a  cost  savings  of  $10.6M  per  year.  The  estimated  $10.6M  savings  is 
only  for  the  Longbow;  the  Patriot  Advanced  Capability-3  (PAC-3)  and  JAVELIN 
programs  are  planned  to  be  tested  in  the  STAF  with  similar  expected  savings. 

Aerial  Cable  Range  (ACR).  The  ACR  at  White  Sands  Missile  Range  (WSMR)  employs 
a  suspended  aerial  cable  system  that  features  highly  repeatable  test  conditions,  reusable 
targets,  and  fast  turnaround  of  tests.  The  heart  of  the  ARC  is  a  3 -mile  long  suspended 
Kevlar  cable  that  serves  as  the  path  for  captive  vehicles  and  supports  test  objects  weighing 
up  to  20,000  pounds.  It  is  the  longest  unsupported  cable  span  in  the  world.  The  initial 
cost  of  this  range  was  $32.2M.  The  recently  completed  tests  of  the  Joint  Missile 
Approach  Warning  System  (MAWS)  resulted  in  a  cost  avoidance  of  $13.8M.  Use  of  full- 
scale  drone  aircraft  would  have  cost  approximately  $14.5M.  The  actual  cost  of  the 
MAWS  test  using  reusable,  captive  targets  on  the  ACR  was  $700,000.  Captive  targets 
are  not  destroyed  during  testing  on  the  ACR.  A  “heat  source”  hangs  below  the  captive 
target  so  that  the  missile  would  aim  at  the  heat  source  and  not  the  targets.  Other 
upcoming  events  include  tests  for  the  Apache  (AH-64)  Target  Characterization  in  June 
1995  and  the  Post  Burnout  testing  of  Infrared  Sensors  in  July  1995. 
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Test  Item  Stimulators  (TIS).  The  TIS  at  Electronic  Proving  Ground  (EPG)  provides 
non-radiating  simulated  digital  message  traffic  input  to  challenge  large  network,  computer- 
based  command,  control,  and  communications  systems.  Through  the  Test  Control  Center, 
inputs  are  time  coded  according  to  the  mission  or  exercise  scenario,  and  are  variable  and 
repeatable,  as  required.  Through  use  of  these  stimulators,  and  the  control  model  for 
operating  the  stimulators  and  test  items,  the  cost  of  testing  the  Enhanced  Position 
Location  Reporting  System  (EPLRS)  was  reduced  by  $2M  (30%).  Savings  were  realized 
through  improved  test  efficiency  and  reduction  of  number  of  operators  required  to  support 
the  testing.  The  TIS  can  be  adapted  to  any  communication  system.  It  was  also  used  for 
Mobile  Subscriber  Equipment  (MSE)  test.  Joint  Tactical  Information  Distribution  System 
(JTIDS)  test,  and  the  Army  Tactical  Command  and  Control  System  (ATCSS)  operational 
test. 

Trajectory  Sense  and  Destroy  Armor  (SADARM)  Simulation.  Real-time  tracking 
data  from  range  radars  are  used  to  model  the  ballistic  simulation  for  the  SADARM 
projectile  trajectory  to  provide  sufficiently  accurate  pointing  data  to  target  area 
instrumentation.  This  real-time  optimal  estimation  software  enables  the  downrange  auto¬ 
trackers  to  acquire  and  track  the  incoming  projectile  and  quickly  transition  to  acquire 
critical  end-game  data  on  SADARM  submunitions  upon  dispensing  from  the  projectile. 
This  new  tracking  capability  has  avoided  $12M  in  instrumentation  cost. 

Patriot  Simulation  Program.  The  testing  of  P AC-3  hardware  and  software  is  supported 
by  several  models  and  simulators  which  reduce  the  requirement  for  launching  targets, 
firing  test  missiles,  and  telemetering  flight  data.  The  PAC-3,  being  in  the  early  stages  of 
the  development,  has  not  entered  U.S.  Government  technical  testing.  It  is  anticipated  that 
once  testing  begins,  a  considerable  amount  of  savings  may  be  realized  through  flight 
mission  simulation  in  lieu  of  actual  firings.  Each  flight  can  be  simulated  numerous  times  to 
increase  the  confidence  level  of  flight  test  performance  without  the  use  of  live  missiles, 
targets,  instrumentation,  and  some  data  reduction  expense.  Although  the  Patriot  modeling 
and  simulation  capabilities  belong  to  either  the  Patriot  Project  Office  or  the  Prime 
Contractor,  TECOM  will  interface  with  these  capabilities  during  anticipated  virtual 
testing.  At  about  $750,000  per  launch  of  a  Patriot  missile,  modeling  and  simulation  are 
essential  to  the  PAC-3  test  program.  In  addition: 

1.  The  Guidance  Test  and  Simulation  Facility  is  a  full  hardware-in-the-loop 
guidance  simulator  for  the  Patriot  system,  providing  endgame  geometry  and 
miss  distance  in  lethality  analyses. 


2.  The  Flight  Mission  Simulator  provides  a  controlled  environment  of  various 
simulated  target  signature  and  electronic  countermeasures  inputs  to  the  system 
surveillance  function,  as  well  as  simulated  missile  responses  for  the  guidance 
function. 

3.  The  Multi-function  Simulation  models  Patriot  search,  tract,  and  engagement 
capabilities  under  radar  loading. 

4.  The  PAC  Simulation  provides  a  high-fidelity  digital  simulation  of  the 
surveillance  function,  missile  dynamics,  and  lethality  function  for  pre-flight 
predictions  and  post-flight  reconstruction  of  flight  tests. 

5.  The  Counter  Anti-Radiation  Missile  is  a  digital  simulation  of  anti-radiation 
missile  performance  against  the  Patriot  system. 

Physical  Simulation  of  Bridge  Crossing.  Traditionally,  bridge  durability  testing  is 
conducted  by  performing  multiple  crossings  of  vehicles.  This  method  requires  12  weeks 
to  conduct  3,000  crossings  and  costs  $325,000.  Bridge  durability  tests  are  now 
conducted  using  a  mixture  of  actual  and  simulated  crossings.  ATC  conducted  the  bridge 
crossing  using  actual  equipment  and  vehicles  that  were  instrumented  to  measure  physical 
characteristics,  and  simulations  were  conducted  using  a  physical  bridge  crossing  simulator 
at  Fort  Belvior,  VA.  This  new  simulated  test  method  reduces  the  time  to  nine  weeks  and 
costs  to  $110,000.  During  future  use  of  the  simulator,  particularly  for  unique  foreign 
bridges,  this  mix  of  live  and  simulated  testing  will  be  conducted  exclusively  at  Aberdeen. 

Vibration  Test  Facility  (VTF).  A  long-established  use  of  simulation  in  testing  is  the 
simulation  of  the  natural  and  induced  environments  to  which  Army  materiel  is  subjected. 
One  such  environment  is  vibration.  Tanks  with  dummy  ammunition  are  driven  around  and 
instrumented  to  measure  the  vibration  levels.  Those  profiles  are  simulated  with  shaker 
tables  at  the  VTF.  It  would  take  sbc  months  of  field  testing  to  subject  tank  ammunition  to 
the  transport  vibration  during  its  typical  life  cycle.  At  $1,200  per  tank  hour,  doing  this 
testing  “live”  would  be  prohibitive.  Instead,  24  hours  of  transport  simulation  on  a  “shaker 
table”  subjects  tank  ammunition  to  expected  life  cycle  transport  vibration.  The  VTF  is 
also  used  to  conduct  tests  where  other  components  (i.e.,  radios,  power  units,  heater  units, 
environmental  conditioning  units,  and  autoloader  equipment)  are  subject  to  various  driving 
conditions.  In  both  applications,  increased  simulation  capabilities  are  reducing  costly  field 
testing. 

Joint  Precision  Strike  Demonstration  (JPSD).  Redstone  Technical  Test  Center 
instrumented  the  Multiple  Launch  Rocket  System  (MLRS)  Launcher,  acquired  weapon 


system  performance  data,  and  analyzed  the  data  from  the  JPSD  mission.  In  addition, 
WSMR  successfully  fired  live  Army  Tactical  Missiles  as  part  of  the  distributed  JPSD, 
using  interactive  live,  constructive,  and  virtual  simulation  at  various  sites  to  develop  an 
Army  sensor-to-shooter,  precision,  deep-strike  capability. 

Combined  Arms  Synthetic  Experiment  (CASE).  The  CASE  is  a  concept  of  linking 
two  test  centers  to  conduct  live,  virtual,  and  constructive  simulations.  This  concept  was 
successfully  demonstrated  in  December  1994.  In  the  CASE  demonstration,  real  weapon 
systems  operating  at  YPG  and  WSMR  were  synthetically  fused  with  a  JANUS 
Warfighting  model,  using  Distributed  Interactive  Simulation  protocols.  Three-dimensional 
displays  were  generated  based  on  the  merged  data  to  provide  high-resolution  visualization 
of  the  experiment  in  near  real  time.  In  addition,  the  injection  of  the  real  entities  into  the 
constructive  simulation  required  the  transformation  of  all  live  entities  onto  the  terrain 
employed  by  the  model.  Pieces  and  modules  of  this  capability  are  being  developed.  This 
capability  should  be  fully  operational  in  1998  as  part  of  the  VPG.  The  CASE  concept 
demonstrated  the  capability  to  support  the  Force  XXI  initiative.  Advanced  Warfighting 
Experiments,  Advanced  Technology  Demonstrations,  and  Joint  Service  Integration 
Exercises.  A  CASE  II  is  planned  that  will  demonstrate  additional  capabilities  and 
highlight  tri-Service  applications  at  the  Major  Range  and  Test  Facility  Base  (MRTFB). 

3-D  Virtual  Test  Range  (VTR).  (ongoing).  The  VTR  is  generate  from  the  Defense 
Mapping  Agency’s  Digital  Terrain  Elevation  Data  Level  II  (30  square  meter  resolution) 
merged  with  weapon  system  models,  human-in-the-loop,  and  hardware-in-the-loop  for 
simulation  of  weapon  system  engagements.  The  VTR  includes  the  Combine  Arms 
Synthetic  Experiment  which  uses  Sensor  Data  Fusion  (SDF).  The  SDF  allows  Virtual 
Reality  Display  of  systems  where  optics  fail,  e.g.,  missile  cruising  at  140,000  ft. 

Integrated  Radar  and  Infrared  Analysis  Modeling  (BRIAM)  (ongoing).  The  IRIAM  is 
a  tri-Service  program.  It  generates  three-dimensional  graphics  for  the  representation  of 
test  data,  real-time  visual  test  reports,  and  flight  test  reports.  The  tri-Services  are 
incorporating  the  Navy  infrared  modeling  software,  Air  Force  aircraft  models  and  weather 
effects  stimuli,  and  Army  virtual  display  and  range  control  software. 

Situational  and  Reality  Display  System  (SARDS)  (ongoing).  The  SARDS  consists  of 
the  VTR  interfaced  to  the  Global  Positioning  System  Time  Space  Position  Information 
equipment  to  provide  real-time  tracking  to  identify  location  and  status  of  selected  aircraft, 
vehicles,  etc. 


C-17  Airdrop  (ongoing).  The  results  of  live  airdrops  at  YPG  are  being  used  to  model  the 
airdrop  characteristics  of  the  C-17  Globemaster  III. 

Dynamic  Infrared  Scene  Projector  (DIRSP)  (ongoing).  The  DIRSP  is  a  new  test 
technology  concept  that  projects  synthetic  infrared  imagery  into  the  sensor  entrance 
aperture.  This  system  and  the  imaging  infrared  sensor  are  analogous  to  television  and  the 
human  eye.  The  DIRSP,  when  integrated  with  existing  models  and  simulation  techniques, 
will  bring  “field  testing”  into  the  laboratory  by  supplying  synthetic  infrared  field  test 
environments  for  operational  evaluation  of  imaging  infrared  sensors,  subsystems,  and 
systems. 

Covert  Remote  Electronic  Warfare  Simulator  (CREWS)  (ongoing).  The  CREWS 
concept  is  based  on  replacing  the  use  of  high  power,  broadband  jammers  to  simulate 
electronics  warfare  threat  systems  with  a  continuous  wave  tone  generator  of  moderate 
power.  This  tone  generator  is  used  to  radiate  a  signal  of  kno\vn  amplitude  in  or  near  the 
frequency  of  jamming  interest.  The  CREWS  is  mounted  in  either  an  airborne  or  ground 
platform  to  represent  the  real  threat.  For  the  JTIDS  analysis,  at  least  three  “good”  data 
points  for  each  jamming  condition  are  required.  On  an  average,  a  field  jamming  test 
requires  three  flights  to  get  a  single  valid  data  point.  A  live  field  jamming  test  for  a  single 
jamming  condition  would  cost  approximately  $52,000.  Using  the  CREWS,  the  aircraft 
will  be  flown  one  time  and  the  CREWS  will  record  the  control  tone’s  received  signal  level 
(RSL);  subsequent  data  points  will  simply  require  playing  back  the  recorded  RSL 
information  through  the  CREWS.  The  CREWS  jamming  test  will  cost  only  $5,000. 
CREWS  modules  are  being  produced  in  FY95  to  instrument  the  Enhanced  Position 
Location  Reporting  System  (EPLRS),  Mobile  Subscriber  Equipment  (MSE),  Joint 
Tactical  Information  Distribution  System  (JTIDS),  and  Single  Channel  Ground  and 
Airborne  Radio. 

Environmental  Issues  Guide  for  Heuristic  Testing  (EIGHT)  (ongoing).  The  EIGHT 
is  a  knowledge  base  of  how  environments  affect  materiel  systems.  The  EIGHT  is  a  unique 
tool  that  looks  at  the  effects  of  environments  on  materiel  systems,  not  effects  of  systems 
on  the  environment.  EIGHT  arranges  information  in  a  four-dimensional  matrix  with  about 
200,000  cells.  Each  cell  consists  of  4  climatic  regions  (e.g.,  cold,  desert,  tropic,  and 
temperate),  50  environmental  factors  in  4  categories  (e.g.,  natural,  induced,  constructed, 
and  battlefield),  10  analysis  areas  (e.g.,  operability,  safety,  vulnerability),  and  100  mission 
or  commodity  areas  (e.g.,  air  defense,  aviation,  command  and  control).  This  database 
provides  ground  truth  for  the  VPG  to  define  interaction  of  the  test  item  and  the 
environment,  and  also  provides  historical  test  results  for  different  commodity  systems  and 
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environmental  conditions.  It,  therefore,  provides  lessons  learned  from  these  tests  so  that 
mistakes  are  not  repeated  and  future  testing  is  better  focused  on  relevant  parameters 
whether  live  or  simulated. 

HELLFIRE  Simulation  (ongoing).  The  Laser  Designator  Weapon  System  Simulation 
(LDWSS)  is  accredited  for  use  to  extrapolate  HELLFIRE  weapon  system  performance 
based  on  subsystem  and  component  level  testing.  The  LDWSS  model  is  being  upgraded 
and  modified  to  allow  for  higher  resolution  (i.e.,  discrete  component)  effects  analysis. 

Target  Acquisition  Model  Improvement  Program  (TAMIP)  (ongoing).  Models 
undergoing  improvement  in  the  TAMIP  include  millimeter-wave,  infrared,  visible,  and 
acoustic  capabilities.  The  enhanced  models  provide  an  objective  means  for  comparing  the 
vulnerability  of  vehicles  as  it  relates  to  detection  on  the  battlefield.  The  models  are  used 
for  quantifying  the  value  added  when  vehicle  signatures  are  reduced,  and  to  drive 
requirements  for  signature  specification  of  future  vehicles. 

Air  Force  AMRAAM  Test  Cost  Savings.  During  the  last  eight  years,  more  than  130 
DT&E  missiles  have  been  launched  from  test  aircraft  against  drone  aircraft.  In  addition, 
approximately  45  FOT&E  missiles  have  been  launched  out  of  a  planned  total  of 
approximately  65  FOT&E  missiles.  Only  8  live  warheads  have  been  used  in  the 
approximately  175  launches  to  date.  The  end-game  effectiveness  model,  SHAZAM,  has 
been  successfully  used  to  predict  and  subsequently  to  assess  the  lethality  of  each  missile  at 
end  game. 

This  has  resulted  in  a  direct  program  savings  of  approximately  $250M  in  drone  aircraft 
costs  alone  because  very  few  of  the  drones  were  damaged  by  warhead  blast  or  fragments 
since  only  eight  live  warheads  were  used.  In  most  cases,  the  $2.5M  drones  were  able  to 
be  returned  safely  and  used  repeatedly.  Table  BIB-2  depicts  the  Return  on  Investment 
(ROI). 


Table  BIB-2.  AMRAAM  Test  Costs  Savings  ROI 


Model  Development: 
Analysis  Support: 
Lethality  Tests: 

4  people  X  2  yr  X  $125K/yr  = 

4  people  X  8  5n’  X  $125K/yr  = 
Range  time  and  people  = 

$L0M 

$4.0M 

+$L5M 

Total  Investment: 

$6.5M 

Direct  Savings: 

$250.0M 

ROI: 

$250M/$6.5M  = 

38 

Other,  less  tangible  benefits  were  as  follows: 
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•  No  more  F-106  aircraft  suitable  to  be  “droned”  remain  at  Davis-Monthan 
AFB.  Drones  are  now  being  made  from  F-4  aircraft  and  it  would  be  difficult  to 
support  a  missile  flight  test  program  if  drones  were  not  returned  for  reuse. 

•  The  environmental  impact  on  the  Gulf  of  Mexico  would  be  significant  if 
hundreds  of  drone  aircraft  with  leaking  fuel  tanks  were  crashed  into  the  water. 
The  use  of  lethality  modeling  with  SHAZAM  precludes  this  environmental 
problem. 

•  In  actuality,  the  use  of  simulation  permits  the  assessment  of  the  entire  end 
game,  including  the  fly-by  which  would  be  denied  if  the  warhead  were 
detonated  and  the  telemetry  ceased  to  be  transmitted.  The  use  of  simulation 
permits  collection  of  these  useful  data  after  the  warhead  would  have  detonated. 

GBU-28  Precision  Guided  Weapon  Test  Savings.  The  GBU-28  was  developed  at 
Eglin  AFB,  FL,  in  less  than  six  weeks  in  response  to  an  urgent  need  during  the  Gulf  War. 
The  short  time  did  not  permit  any  of  the  normal  testing  which  usually  accompanies  any 
weapon  development  program.  Consequently,  the  program  management  relied  almost 
exclusively  on  lethality  and  vulnerability  modeling  to  design  and  predict  the  performance 
of  the  new  weapon  system 

The  depth  of  penetration  into  subterranean  structures  was  calculated  using  anal54;ically 
derived  penetration  equations.  One  drop  test  of  an  inert  GBU-28  was  conducted  on  one 
type  of  soil  to  check  the  model;  however,  the  primary  purpose  of  the  drop  test  was  to 
verify  the  structural  integrity  of  the  steel  case.  The  performance  of  the  GBU-28,  the  depth 
of  penetration,  the  lethality  of  the  warhead  in  a  new  case,  and  the  vulnerability  of  the 
targets  were  predicted  successfully  by  lethality  and  vulnerability  models.  Without  any 
prior  drops  of  a  live  GBU-28,  live  GBU-28  weapons  were  dropped  during  the  air  war 
based  on  the  confidence  in  current  lethality  and  vulnerability  models. 

Mk  Series  Bomb  Fragment  Data.  The  Mk-84  general  purpose  bomb  was  retrofitted  by 
the  U.S.  Air  Force  to  have  the  option  for  either  tail  or  nose  fuzing  or  both.  Arena  tests 
were  conducted  at  a  cost  of  more  than  $400K  to  determine  the  difference  in  fragment 
spray  patterns  resulting  from  nose  and  tail  fuzing.  Based  on  these  tests,  lethality  modeling 
techniques  were  developed  to  model  the  lethal  fi'agment  spray  pattern.  This,  in  turn, 
permitted  both  the  Mk-82  and  the  Mk-83  general  purpose  bomb  fragment  spray  patterns 
to  be  characterized  without  the  expenditure  of  nearly  $900K  for  additional  arena  tests. 
Table  BIB-3  depicts  the  ROI. 


Table  BIB-3.  Mk  Series  Bomb  Fragment  ROI 


Model  Development; 
Analysis  Support: 

Used  JMEMS  (sunk  cost)  = 

2  people  X  .33  yr  x  $125K/yr  = 

$0.0K 

+$82.5K 

Total  Investment: 

Direct  Savings: 

ROI: 

$82.5K 

$900.0K 

$900K/$82.5K  - 

11 

Determination  of  BLU-109  Performance  Envelopes.  When  the  BLU-109  was  under 
development  by  the  U.S.  Air  Force,  it  was  necessary  to  conduct  numerous  drop  tests  of 
the  weapon  from  an  aircraft  to  determine  the  limits  of  impact  angle,  impact  speed,  and 
angle  of  attack  which  would  permit  successful  penetration  (without  ricochet  or  partial 
penetration)  of  concrete  surfaces  such  as  runways.  More  than  80  tests  at  approximately 
$5  OK  per  drop  were  planned  to  obtain  the  required  data. 

Program  analysts  recommended  the  use  of  an  analytical  technique  to  reduce  the  number  of 
drops  from  80  to  only  15  while  still  providing  the  required  data  to  assess  the  lethality  of 
the  BLU-109  warhead  throughout  the  performance  envelope.  This  use  of  lethality 
modeling  resulted  in  a  one-time  cost  savings  of  more  than  $3M  on  this  one  weapon 
system.  Table  BIB-4  depicts  the  ROI. 


Table  BIB-4.  BLU-109  Performance  Envelope 


Model  Development: 
Analysis  Support: 

Integral  with  analysis  = 

2  people  X  0.33  x  $125Kyr  = 

$0.0K 

+$82.5K 

Total  Investment: 

Direct  Savings; 

ROI: 

$82.5K 

$3.0M 

$3.0M/$82.5K  = 

36 

[TILV  1995] 

DoD  FY95  Master  Plan  for  Target  Interaction,  Lethality  and  Vulnerability  (TILV) 
Science  and  Technology  (S&T)  Programs.  Volume  I:  Classical  Ballistic  Threats.  U.S. 
Department  of  Defense,  Washington,  DC,  May  4,  1995. 

Air-to-Air  Missile  Engagement  Analysis.  Beginning  in  1980  the  Air  Force  conducted 
an  air-to-air  missile  engagement  analysis  called  ACEVAL/AIMVAL  which  involved 


multiple  aircraft  in  mock  aerial  duels.  This  effort  cost  more  than  $50M  and  required  more 
than  three  years  to  complete. 

A  few  years  later,  the  AMRAAM  Program  required  similar  but  additional  data  on  the 
effectiveness  of  AMRAAM  when  used  in  conjunction  with  short-range  missiles  in  aerial 
combat  situations.  Program  analysts  successfully  convinced  the  program  management  that 
the  required  data  could  be  obtained  without  the  costly  flight  tests  by  using  lethality 
modeling  combined  with  aerial  engagement  modeling. 

As  a  result,  the  AMRAAM  Operational  Utility  Effectiveness  analysis  was  conducted  for 
$20M  in  two  years,  a  savings  of  $30M  and  one  year.  One  added  advantage  was  that  three 
times  as  many  engagements  were  analyzed  than  would  have  been  done  in  mock  combat. 
Table  BIB-5  depicts  the  ROI. 


Table  BIB-5.  Air-to-Air  Missile  Engagement  Analysis  ROI 


Model  Development: 

Analysis  Support: 

Single  cost  includes  both  model 
development  and  analysis  over  a 
two-year  period 

$20.0M 

Total  Investment: 

$20.0M 

Direct  Savings: 

Test  costs  not  incurred  = 

$50.0M 

200  people  x  1  yr  x  $125K/yr  = 

$25.0M 

Total  Direct  Savings: 

$75.0M 

ROI: 

$75.0M/$20.0M  = 

4 

Wide  Area  Anti-armor  Munition  (WAAM).  The  Air  Force  conducted  developmental 
work  on  this  system  in  the  early  1980s  but  did  not  continue  into  full-scale  development  or 
production  because  lethality  analyses  showed  that  the  system  could  not  obtain  the  required 
kills  per  pass  to  make  the  system  cost  effective.  The  cancellation  of  this  program  was 
based  on  lethality  analysis  and  resulted  in  estimated  cost  savings  of  $30M  for  the  planned 
full-scale  development  effort.  Table  BIB-6  depicts  the  ROI. 


Table  BIB-6.  Wide  Area  Anti-armor  Munition  ROI 


Model  Development: 
Analysis  Support. 

Integral  with  analysis  = 

6  people  X  1  yr  X  $125K/yr  = 

SO.OOM 

+$0.75M 

Total  Investment: 

Direct  Savings: 

ROI: 

$0.75M 

$30.00M 

$30M/$0.75M  = 

40 

BIB-59 


Hypervelocity  Missile.  The  Hypervelocity  Missile  program  was  terminated  in  the  early 
stages  of  full-scale  development,  based  on  an  analysis  using  lethality  models  which 
indicated  that  the  payoff  would  not  be  as  great  as  expected  during  the  initial  planning 
effort.  The  cancellation  of  this  effort  was  based  solely  on  a  lethality  analysis  and  resulted 
in  an  estimated  cost  savings  of  $10M  for  the  remaining  full-scale  development  effort. 
Table  BIB-7  depicts  the  ROI. 


Table  BIB-7.  Hypervelocity  Missile  ROI 


Model  Development: 
Analysis  Support: 

Integral  with  analysis  = 

4  people  X  1  yr  X  $125K/yr  = 

SO.OOM 

+$0.50M 

Total  Investment: 

Direct  Savings: 

ROI: 

$0.50M 

SIO.OOM 

$10M/$0.50M  = 

45 

Intelligence  Shelter  Attack  Submunition  (ISAS).  The  German  government  proposed  a 
joint  development  effort  with  the  United  States  to  develop  ISAS  in  1987  for  a  total 
development  cost  of  approximately  $80M.  After  setting  up  a  joint  program  office  at  Eglin 
AFB,  FL,  and  signing  the  appropriate  agreements,  a  detailed  cost-effectiveness  analysis 
was  conducted  using  lethality  models  to  determine  the  payoff  of  the  ISAS  to  the  U.S.  Air 
Force  if  it  were  developed.  The  results  of  the  analysis  clearly  showed  that  the  system 
would  not  perform  under  all  conditions  and  would  not  be  advantageous  to  the  Air  Force. 
Thus  the  program  was  canceled  solely  upon  the  results  of  a  lethality  analysis  with  a 
resultant  savings  of  $40M,  the  Air  Force  share  for  the  proposed  development  effort. 
Table  BIB-8  depicts  the  ROI. 


Table  BIB-8.  Intelligence  Shelter  Attack  Submunition  ROI 


Model  Development: 
Analysis  Support: 

Integral  with  analysis  = 

4  people  X  1 5n'  X  $125K/yr  = 

SO.OOM 

+$0.50M 

Total  Investment: 

Direct  Savings: 

ROI: 

$0.50M 

SIO.OOM 

$10M/$0.50M  = 

45 

Kinetic  Energy  Penetrator  (KEP).  The  U.S.  Air  Force  developed  a  working  version  of 
a  new  runway  penetrator  called  the  Kinetic  Energy  Penetrator  (KEP)  which  was  ready  for 
production  in  1980.  At  the  request  of  the  program  management,  a  tiger  team  was 


convened  to  determine  the  suitability  of  the  system  for  production.  After  a  review  of  the 
program  and  the  completion  of  a  detailed  effectiveness  analysis  using  lethality  models  of 
the  system,  the  program  was  canceled.  The  technology  was  set  on  the  shelf  and  the  Air 
Force  saved  a  minimum  of  $50M  in  production  costs  which  were  never  incurred.  Table 
BIB-9  depicts  the  ROI. 


Table  BIB-9.  Kinetic  Energy  Penetrator  (KEP) 


Model  Development: 
Analysis  Support; 

Range  Tests: 

8  people  X  3  mo  X  $125K/yr  = 

16  people  X  3  mo  X  $125K/yr  = 
Range  time  and  people  = 

$0.25M 

$0.50M 

+$0.30M 

Total  Investment: 

Direct  Savings; 

ROI: 

$0.75 

$40.00M 

$40M/$0.75M  = 

53 

JP-233  Runway  Attack  Munition.  The  JP-233  was  a  joint  development  effort 
conducted  with  the  United  Kingdom  during  the  late  1970s.  After  spending  several 
millions  in  the  development  costs  with  the  United  Kingdom,  the  program  was  finally  ready 
for  production  in  the  early  1980s.  Before  commitment  to  production,  the  U.S.  Air  Force 
commissioned  a  study  to  determine  the  cost  effectiveness  of  the  JP-233  for  Air  Force  use. 
The  resulting  analysis  using  both  lethality  and  vulnerability  (LfV)  models  clearly  showed 
that  the  system  was  not  cost  effective  for  Air  Force  use.  The  use  of  LA^  modeling  in  this 
case  resulted  in  the  U.S.  withdrawal  from  the  program  and  saved  the  Air  Force  more  than 
$50M  in  production  costs.  Table  BIB-10  depicts  the  ROI. 


Table  BIB-10.  JP-233  Runway  Attack  Munition 


Model  Development: 
Analysis  Support: 

Range  Tests: 

4  people  X  3  mo  X  $125K/yr  = 

20  people  ( 3  mo  ( $125K/yr  = 
Range  time  and  people  = 

$0.15M 

$0.65M 

+$0.30M 

Total  Investment: 

Direct  Savings: 

ROI: 

Sl.lOM 

$54.00M 

$54M/$1.10M  = 

49 

Boosted  Kinetic  Energy  Penetrator  (BKEP).  After  withdrawing  from  the  JP-233 
program,  the  U.S.  Air  Force  spent  several  years  and  millions  of  dollars  developing  its  own 
runway  defeat  munition,  called  BKEP.  After  completion  of  full-scale  development  in 


1987,  continuing  analysis  of  the  BKEP’s  effectiveness  by  Air  Force  analysts  using  runway 
vulnerability  models  resulted  in  the  determination  that  the  BKEP  would  not  be  able  to 
defeat  some  of  the  new  runway  designs  being  installed  at  the  Warsaw  Pact  air  bases. 
Based  primarily  on  the  results  of  this  vulnerability  analysis,  the  BKEP  program  was 
canceled  with  an  estimated  $130M  savings  in  production  costs  alone.  Table  BIB-11 
depicts  the  ROI. 


Table  BIB-11.  Boosted  Kinetic  Energy  Penetrator  ROI 


Model  Development: 
Analysis  Support: 

Range  Tests: 

4  people  ( 6  mo  (  $125K/yr  = 

12  people  ( 1  yr  ( $125K/yr  = 
Range  time  and  people  = 

$0.25M 

$1.50M 

+$1.00M 

Total  Investment: 

Direct  Savings: 

ROI: 

$2.75M 

$130.00M 

$130M/$2.75M  = 

47 

JAVELIN  Antitank  Guided  Missile.  In  the  mid-1980s,  the  Army  initiated  development  of 
a  new  man-portable  antitank  guided  missile  to  counter  recent  changes  in  the  armor  threat. 
Soviet  tanks  had  become  more  heavily  armored,  and  it  was  clear  that  a  new  warhead 
technology  was  needed,  and  that  an  unconventional  flight  trajectory  was  required  to 
deliver  the  warhead  against  the  “softer”  portions  of  a  target.  Without  these  advanced 
technologies,  it  was  not  possible  to  achieve  the  desired  lethality  within  acceptable  weight 
constraints. 

Using  flight  simulation  models  and  terminal  lethality  models,  various  delivery  strategies 
and  warhead  technologies  were  evaluated.  In  this  way,  the  best  design  approach  was 
identified  without  destructive  testing.  Further,  the  required  warhead  size  was  determined 
by  parametrically  varying  the  size  and  computing  kill  probabilities. 

By  contrast,  when  the  U.S.  Army  started  development  of  its  first  antitank  guided  missile, 
the  Shillelagh,  it  was  not  known  what  warhead  size  was  needed  to  defeat  the  threat  tanks 
of  the  day,  and  no  models  were  available  to  address  the  question  analytically.  It  was 
necessary  to  conduct  a  rather  large  experiment  in  which  a  scaled  family  of  warheads  were 
fired  against  tanks  to  observe  the  lethality  as  a  function  of  size.  One  hundred  twenty 
warheads  were  fired.  If  it  were  necessary  to  conduct  such  an  experiment  today,  the  cost 
would  be  in  the  neighborhood  of  $10M  plus  the  cost  of  the  target  tanks.  Thus,  it  is 


conservatively  estimated  that  analytical  simulation  for  the  man-portable  anti-tank  guided 
missile  saved  at  least  $10M.  Table  BIB- 12  depicts  the  ROI. 


Table  BIB-12.  JAVELIN  Antitank  Guided  Missile  ROI 


Model  Development. 
Analysis  Support: 

1  person  ( 6  mo  (  $125K/yr 

6  people  (  9  mo  (  $125K/yr  = 

$0.06M 

+$0.56M 

Total  Investment: 

Direct  Savings: 

Total  Direct  Savings: 

ROI: 

Test  costs  = 

Tanks  saved  =  8  x  $0.50M  = 

$0.62M 

SIO.OOM 

$4.00M 

$14.00M 

$14M/$0.62M  = 

23 

M2  Bradley  Fighting  Vehicle  System.  When  the  U.S.  Army  began  development  of  the 
M2  Bradley  Fighting  Vehicle  System,  it  faced  a  unique  problem.  The  Army  had  never  had 
a  vehicle  of  this  type,  i.e.,  a  vehicle  which  both  transports  an  infantry  squad  and  possesses 
significant  firepower.  While  the  intended  tactical  role  of  the  vehicle  was  defined,  there 
was  no  basis  in  experience  for  selecting  the  vehicle  characteristics  that  would  best  serve 
the  planned  tactical  role.  The  only  experimental  approach  to  the  problem  was  to  design 
and  construct  several  prototypes  and  subject  them  to  field  testing  at  a  cost  of  tens  of 
millions  of  dollars. 

Using  analysis  techniques,  it  was  possible  to  address  the  issues  analytically  and  save  nearly 
all  of  this  cost.  Preliminary  design  concepts  were  sought  from  industry  and  from 
government  with  different  design  features.  Each  of  these  designs  was  evaluated 
analytically  in  terms  of  vulnerability,  mobility,  firepower  characteristics,  estimated  cost, 
and  logistic  burden.  With  the  results  of  these  evaluation  as  inputs,  the  combat 
effectiveness  of  each  design  was  estimated  by  use  of  a  combat  simulation  model.  The 
results  of  these  evaluations  allowed  the  Army  to  select  the  best  vehicle  for  engineering 
development  on  a  stronger  basis  than  had  been  possible  on  any  previous  occasion  and  at 
an  estimated  savings  of  tens  of  millions  of  dollars.  Table  BIB- 13  depicts  the  ROI. 


Table  BIB-13.  M2  Bradley  Fighting  Vehicle  ROI 


Model  Development: 
Analysis  Support; 

2  people  X  6  mo  X  $125K/yr 

6  people  X  1  yr  X  $125K/yr  = 

$0.1 3M 

+$0.75M 

Total  Investment; 

Direct  Savings: 

ROI: 

$0.88M 

$30.00M 

$30M/$0.88M  = 

34 

Abrams  M1A2  Tank  Vulnerability  Testing.  The  requirement  for  ballistic  vulnerability 
testing  on  fully  operational  Abrams  tanks  was  reduced  by  conducting  vulnerability 
computer  modeling  to  predict  damage  from  penetrating  threats  before  “Controlled 
Damage”  tests  were  conducted.  In  effect,  the  computer  predictions  permitted  the 
damaged  components  to  be  identified  and  disabled  in  the  Controlled  Damage  tests,  and  the 
tests  permitted  the  degradation  of  the  system  in  combat  to  be  determined  without  damage 
to  the  total  Abrams  vehicle.  Costs  for  repair  of  ballistic  damage  to  one  Abrams  can  run  as 
high  as  $1.5  million  per  shot  if  damage  were  catastrophic.  For  a  series  of  20  Controlled 
Damage  tests,  the  total  cost  was  $80K,  compared  with  a  potential  loss  of  $30M  for  20 
tests  on  the  full-scale  Abrams.  The  20  Controlled  Damage  tests,  based  on  computer 
simulations  to  predict  damage,  cost  only  $80K,  which  was  a  savings  of  S29M.  Table  BIB- 
14  depicts  the  ROI. 


Table  BIB-14.  Abrams  Ml  A2  Vulnerability  ROI 


Model  Development; 
Analysis  Support: 

Controlled  Damage  Tests: 

6  people  X  1  yr  X  $125K/yr  = 
20  tests  X  $4K/test  = 

$1.00M 

$0.75M 

+$0.008M 

Total  Investment: 

Direct  Savings: 

ROI: 

$1.83M 

$30.00M 

$30M/$1.83M  = 

16 

Block  3,  M1A2  Abrams  Tank.  During  the  development  of  the  Block  3,  M1A2  Abrams 
tank,  an  analysis  of  the  radical  front-engine  tank  design  was  conducted  to  assess  the 
vulnerability  of  the  design.  The  results  of  the  analysis  clearly  showed  the  new  design  to  be 
very  vulnerable  to  certain  threats  which  Desert  Storm  data  proved  to  be  correct.  As  a 
result,  a  very  costly  development  program  was  terminated  with  a  savings  conservatively 
estimated  to  be  more  than  SIOOM  for  development.  Table  BIB- 15  depicts  the  ROI. 


Table  BIB-15.  Block  3,  M1A2  Abrams  Tank  Design  ROI 


Model  Development: 

3  people  X  1  yr  X  $125K/yr 

$0.38M 

Analysis  Support: 

7  people  X  1  yr  X  $125K/yr  = 

$0.88M 

Field  Tests: 

Range  time  and  people  = 

+$0.05M 

Total  Investment: 

$1.76M 

Direct  Savings: 

SIOO.OOM 

ROI: 

$100M/$1.76M  = 

57 

Standard  Missile  (SM-2  BLK  IIIA)  DT&E/OT&E  Flight  Tests.  The  Navy  was 
directed  to  reduce  costs  by  $47M  on  the  planned  flight  tests.  This  required  the  elimination 
of  13  missile  flight  tests  out  of  the  planned  30  tests.  As  a  result,  the  Navy  was  required  to 
upgrade  the  existing  COVART  lethality  model  and  the  end-game  effectiveness  model 
called  WHDEVAL  to  evaluate  the  performance  of  the  SM-2  at  regions  of  the  performance 
envelope  which  could  not  be  tested.  The  missile  system  was  subsequently  accepted  based 
on  the  reduced  number  of  flight  tests  and  an  increased  amount  of  simulation  and  analysis 
using  the  above  lethality  and  vulnerability  models.  Table  BIB- 16  depicts  the  ROI. 


Table  BIB-16.  Standard  Missile  (SM-2  BLK  IIIA)  ROI 


Model  Development: 
COVART: 

WHDEVAL: 

Analysis  Support: 
Lethality  Tests: 

2  people  X  2  yr  X  $125K/5n‘  = 

3  people  X  2  yr  X  $125K/yr  = 

3  people  X  2  yr  X  $125K/yr  = 

2  people  X  1  3n'  X  $125K/yr  = 

$0.50M 

$0.75M 

$0.75M 

+$0.25M 

Total  Investment: 

Direct  Savings: 

ROI: 

$2.25M 
$47.00M 
$47M/$2.25M  = 

21 

Phalanx  Close  in  Weapon  System  (CIWS)  Block  0,  DSARC  m  Decision.  After 
spending  more  than  $125M  on  development  of  the  Phalanx  Naval  gun  system,  the  U.S. 
Congress  was  ready  to  cancel  the  program  based  on  the  first  test  results  in  1975.  Since  a 
new  gun  system  design  would  cost  at  least  $125M  at  that  time.  Congress  was  persuaded 
to  delay  program  cancellation  until  the  necessary  analytical  models  were  developed  to 
evaluate  the  Phalanx  performance  under  all  conditions  without  extensive  hardware  testing. 
Within  three  years,  three  models  were  developed  and  validated  based  on  extensive 
laboratory  lethality  tests.  The  resulting  analyses  were  of  sufficient  quality  to  convince 
Congress  to  reinstate  the  original  Phalanx  program,  based  primarily  on  the  results  of 


analytical  modeling  and  simulation  at  a  minimum  savings  of  $125M.  Table  BIB- 17  depicts 
the  ROI. 


Table  BIB-17.  Phalanx  CIWS  Block  0  ROI 


Model  Development: 

Multiple  Plate  Pen  Model; 
Probability  of  Hit  Model: 
Gun  Effectiveness  Model: 
Analysis  Support: 

Lethality  Tests: 

3  people  X  2  yr  X  $125K/yr  = 

2  people  X  2  yr  X  $125K/yr  = 

3  people  X  2  yr  X  $125K/yr  = 

3  people  X  3  yr  X  $125K/yr  = 
people  and  test  assets  = 

$0.75M 

$0.50M 

$0.75M 

$1.12M 

+$5.00M 

Total  Investment; 

Direct  Savings: 

ROI: 

$8.I2M 
$125.00M 
$125M/$8.12M  = 

15 

Phalanx  Close  in  Weapon  System  (CIWS)  Upgrade  Tradeoff  Study.  Within  10  years 
after  the  initial  operational  capability  (IOC)  of  the  Phalanx,  the  advent  of  new  threats 
required  either  development  of  a  new  CIWS  or  improvements  in  the  original  Phalanx 
capability.  By  using  upgraded  version  of  the  existing  lethality  models  developed  for  the 
original  Phalanx  CIWS,  an  in-depth  analysis  of  the  various  options  proved  analytically  that 
upgrades  to  the  original  Phalanx  design  would  provide  the  performance  to  defeat  the 
improved  threat  without  extensive  field  testing.  The  consequences  of  this  analysis  were  to 
eliminate  more  than  $200M  in  design  and  production  costs  of  a  new  system.  Table  BIB- 
18  depicts  the  ROI. 


Table  BIB-18.  Phalanx  CIWS  Upgrade  Tradeoff  Study  ROI 


Model  Development: 

Multiple  Plate  Pen  Model: 
Probability  of  Hit  Model: 
Gun  Effectiveness  Model; 
Analysis  Support: 

Additional  Lethality  Tests: 

3  people  X  1  yr  X  $125K/yr  = 

2  people  X  1  yr  X  $125K/yr  = 

2  people  X  1  yr  X  $125K/yr  = 

4  people  X  2  yr  X  $125K/yr  = 
people  and  test  assets  = 

$0.38M 

$0.25M 

$0.25M 

$0.75M 

+$5.00M 

Total  Investment; 

Direct  Savings; 

ROI: 

$6.63M 
$200.00M 
$200M/$6.63M  = 

30 

AIM-7P  Sea  Sparrow  Ship  Launches.  The  Navy  developed  a  version  of  the  AIM-7  for 
ship  launch  against  specific  threats.  Initially,  10  launches  were  conducted  successfully  out 
of  a  planned  50  launches.  The  cost  of  each  launch  was  $150K  for  the  missile  and  $250K 


for  the  actual  test;  or  a  total  of  $400K  each.  By  using  SCAN,  the  Navy’s  end-game 
effectiveness  model,  to  predict  the  lethality  of  the  missile  for  the  40  remaining  flight  tests 
the  Navy  was  able  to  eliminate  $16M  in  flight  test  costs.  In  addition,  the  use  of  lethality 
analysis  instead  of  actual  flight  tests  reduced  the  total  time  required  to  put  the  AIM-7P 
into  production.  Table  BIB- 19  depicts  the  ROI. 


Table  BIB-19.  AIM-7P  Sea  Sparrow  ROI 


Model  Development: 
Analysis  Support: 
Laboratory  Tests: 

2  people  X  6  mo  X  $125K/yr  = 

3  people  X  1  yr  X  $125K/yr  = 
people  and  hardware  = 

$0.1 2M 
$0.3  8M 
+$0.20M 

Total  Investment: 
Direct  Savings: 

ROI: 

40  tests  at  $400K  each  = 

$0.70M 
$16.00M 
$16M/$0.7M  = 

23 

Phoenix  Missile  6  vs.  6  Tests.  The  Navy  conducted  dozens  of  live  firings  of  the  Phoenix 
air-air  missile  during  development  and  operational  testing  of  the  system.  With  a 
production  missile  cost  of  $2M  each,  a  single  live  firing  against  one  drone  costs  in  excess 
of  $2.5M.  Multiple  firings  of  six  Phoenix  missiles  against  six  target  drones  simultaneously 
cost  in  excess  of  $14M.  In  recent  years  SCAN,  the  Navy’s  end-game  effectiveness  model, 
has  been  used  to  negate  the  need  for  additional  6  vs.  6  live  firing  tests  every  time  a  new 
threat  is  encountered  or  when  the  Phoenix  missile  has  a  new  upgrade.  Over  many  years, 
lethality  modeling  has  been  successfully  used  five  times  for  6  vs.  6  analysis,  with  a  savings 
in  test  costs  on  the  order  of  $70M.  Table  BIB-20  depicts  the  ROI. 


Table  BIB-20.  Phoenix  Missile  6  vs.  6  Tests  ROI 


Model  Development: 
Analysis  Support: 
Laboratory  Tests: 

2  people  X  2  yr  X  $125K/5n' 

5  tests  3  people  x  4  mo  x  $125K/yr  = 

$0.50M 

$0.63M 

+$1.10M 

Total  Investment: 
Direct  Savings: 

ROI: 

$2.23M 
$70.00M 
$70M/$2.23M  = 

31 

Electronic  Countermeasures  (ECM)  Against  Navy  AMRAAM.  The  Navy  has 
conducted  hundreds  of  flight  tests  of  ECM  against  AMRAAM  at  $700,000  per  sortie  with 
approximately  10  intercepts  per  sortie.  This  costly  procedure  has  been  largely  replaced  by 
using  a  modified  version  of  SCAN,  the  Navy’s  end-game  effectiveness  model,  to  evaluate 
the  performance  of  AMRAAM  when  subjected  to  numerous  types  of  ECM  waveforms. 
Athough  some  flight  tests  are  still  conducted,  thousands  of  engagement  simulations  using 


the  SCAN  model  are  now  used  routinely  to  replace  flight  test.  The  savings  are  in  the 
millions  of  dollars  per  year  and  are  expected  to  continue  yearly  as  new  ECM  techniques 
are  evaluated.  Table  BIB-21  depicts  the  ROI. 

Table  BIB-21.  ECM  Against  Navy  AMRAAM  ROI 


Model  Upgrade: 

Analysis  Support; 

ECM  Support  from  Lab; 

1  person  x  1  yr  x  $125K/yr 

2  people  X  1  yr  X  $125K/yr  = 

SO.ISM 

$0.25M 

+$0.20M 

Total  Investment; 

Direct  Savings  (1  yr); 

ROI: 

1 50  missions  x  $70,000  ea  = 

$0.58M 

$10.50M 

$10.5M/$0.58M  = 

18 

[Tillson  et  al.  1992] 

J.  C.  Tillson,  M.  L.  Roberson,  and  S.  A.  Horowitz.  Alternative  Approaches  to 
Organizing,  Training  and  Assessing  Army  and  Marine  Corps  Units.  Part  II:  The  Reserve 
Component.  Institute  for  Defense  Analyses,  Alexandria,  VA,  November  1992. 

There  are  two  goals  for  this  two-part  study.  Part  I  seeks  to  develop  alternative 
approaches  to  organizing  the  Active  Component  of  the  U.S.  Army  and  the  U.S.  Mtirine 
Corps  to  allow  both  Services  to  maintain  force  structure  and  training  readiness  despite 
anticipated  reductions  in  resources  and  operating  tempo.  In  Part  II,  the  goal  is  to  develop 
and  analyze  alternative  ways  of  organizing  and  training  Reserve  combat  forces  that  will 
allow  them  to  better  serve  the  nation’s  diverse  needs.  The  study  also  addresses  changes  in 
the  world  that  make  this  study  important,  such  as  the  shape  of  the  future  battlefields  and 
the  implications  these  battlefields  will  have  on  future  training  and  organizing  of  U.S. 
forces.  Chapter  7,  “New  Approaches  to  Using  Simulation  for  Training,”  and  Chapter  8, 
“New  Approaches  to  Using  Simulators  for  Training”  are  the  most  interesting  to  the 
modeling  and  simulation  community.  These  chapters  describe  a  new  approach  to  using 
virtual  and  constructive  simulation  for  training  the  Army  National  Guard  (ARNG)  and 
U.S.  Marine  Corps  Reserve  (USMCR)  combat  maneuver  units.  It  adapts  current 
simulations  to  training  for  the  Active  Component  to  geographically  removed  Reserve  and 
National  Guard  units  by  applying  new  technologies  available  for  distance  learning.  This 
linkage  for  distance  learning  allows  training  as  a  cohesive  unit.  Previously,  geographically 
separated  units  had  virtually  no  options  available  to  them  if  they  desired  to  leverage  the 
Active  Component  simulation  exercises.  Now,  ARNG  and  USMCR  units  are  able  to 
experience  the  same  staff  training  exercises  available  to  the  Active  Component. 


[Tumage  and  Bliss  1990] 

J.  J.  Tumage  and  J.  P.  Bliss.  An  Analysis  of  Skill  Transfer  for  Tank  Gunnery 
Performance  Using  TOPGUN,  VIGS,  and  ICOFT  Trainers.  ARI  Technical  Report  916. 
USA  Research  Institute  for  the  Behavioral  and  Social  Sciences,  Alexandria,  VA,  1990. 

Researchers  studied  the  transfer  of  training  from  two  part-task  trainers  to  a  full-fidelity 
gunnery  simulator.  Significant  performance  transfer  was  found  from  both  part-task 
trainers  and  full-task  simulators.  Research  demonstrated  how  different  gunnery  trainers 
may  be  used  separately  or  together  to  improve  gunnery  skills  and  to  provide  support  for 
using  less  expensive  part-task  devices  for  training  basic  gunnery  skills. 


[Watson  and  Cooper  1988] 

B.  A.  Watson  and  R.  B.  Cooper.  Relating  Tank  Gunnery  Performance  to  Operational 
Effectiveness  (Study  Summary).  USA  TRADOC  Systems  Analysis  Activity,  White  Sands 
Missile  Range,  NM,  1988. 

No  abstract  available. 


[Wilhoite  1993] 

B.  K.  Wilhoite.  “Bytes  vs.  Bullets:  Crew-Served  Weapons  Simulation  Based  Training.” 
Proceedings  of  the  15th  Interservice/Industry  Training  Systems  and  Education 
Conference,  Orlando,  FL,  December  1993,  475-480. 

Wilhoite  describes  the  ongoing  evaluation  of  the  Indoor  Simulated  Marksmanship  Trainer 
(ISMT),  an  advanced  small-arms  trainer  capable  of  providing  simulation  training  for 
individuals  and  teams  for  every  weapon  in  a  U.S.  Marine  Corps  infantry  battalion. 
Students  who  were  able  to  use  Precision  Gunnery  Training  System  (PGTS)  in  addition  to 
traditional  instmction  generally  performed  better  than  those  receiving  traditional  training 
alone.  Significant  cost  savings  are  possible  as  simulation  is  used  in  lieu  of  live  fire  during 
training. 


BIB-69 


[Winsch  et  al.l994] 

B.  J.  Winsch,  N.  K.  Atwood,  A.  R.  Sawyer,  K.  A.  Quinkert,  C.  K.  Heiden,  P.  G.  Smith, 
and  B.  Schwartz.  Innovative  Training  Concepts  for  Use  in  Distributed  Interactive 
Simulation  Environments.  ARI  Research  Product  94-16.  USA  Research  Institute  for  the 
Behavioral  and  Social  Sciences,  Alexandria,  VA,  1994. 

This  report  describes  innovative  approaches  for  conducting  training  using  emerging 
simulation  technology.  It  is  intended  as  a  forward-looking  reference  for  training 
developers  and  trainers  interested  in  conducting  specific  types  of  training  exercises  using 
Distributed  Interactive  Simulations  (DIS).  The  report  presents  five  innovative  training 
concepts  that  can  be  structured  within  a  DIS  environment  for  delivery  of  five  types  of 
training  exercises,  including  a  Battle  Staff  Planning  Exercise,  a  Leader’s  Reconnaissance 
Exercise,  a  Mission  Rehearsal  Exercise  (Electronic  Sandtable),  a  Mission  Execution 
Exercise  (Electronic  Sandbox),  and  an  Information  Management  Exercise.  The  report 
concludes  with  a  description  of  the  implementation  of  one  of  these  five  concepts,  an 
Information  Management  Exercise  (IMEX). 
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